The term "AI governance" is now used to describe everything from corporate ethics policies to prompt filtering to regulatory compliance frameworks. That breadth has made it harder to reason precisely about what governance actually means when an AI agent is running in production, has credentials, and is taking actions that have real-world effects. This post is an attempt to ground the concept at the runtime layer, starting from first principles rather than from vendor terminology.
By the end, you should have a clear model for what you are trying to enforce, why, and what primitives you need to enforce it. Whether you use Runta or build your own tooling, the underlying structure is the same.
What Is Governance, Precisely?
Governance, in general, is the set of mechanisms that ensure a system operates within defined boundaries and that its operation is accountable. Applied to an AI agent runtime, that breaks into two components:
Boundary enforcement: The agent cannot do things outside the scope of what has been explicitly authorized. The authorization is defined in policy. The enforcement mechanism checks every action against that policy before the action executes.
Accountability: Every action the agent takes is recorded in a form that can be reviewed, audited, and used to reconstruct what happened. The record is complete, accurate, and cannot be altered after the fact.
These two components are independent but complementary. A system with strong boundary enforcement but no accountability log can tell you whether the agent stayed in bounds, but cannot tell you anything about what it did within those bounds. A system with complete logging but no enforcement tells you everything that happened but could not have stopped anything. Both are necessary.
The Four Properties Governance Must Enforce
Working from that definition, a runtime governance framework must enforce four properties:
Isolation: The agent's execution environment is separated from other agents, from the host system, and from any resources not explicitly included in its authorization. Isolation at the process level means that an agent running code cannot affect the filesystem, environment variables, or network interfaces of the host beyond its defined surface. Without isolation, the authorization in policy is effectively unenforceable: a determined agent that can write to arbitrary paths or make arbitrary network connections can circumvent any access control that operates at a higher level.
Containment: The agent's resource consumption is bounded. CPU, memory, network bandwidth, and LLM token budget are all finite resources. Without containment, a single agent run can consume resources that affect other workloads or generate costs that accumulate without bound. Containment is the quota layer: not just "does the agent have permission to call this API" but "how much can it consume in total across all calls in this run."
Access control: The agent can only reach resources that are explicitly authorized. Egress allowlisting is the most critical access control for network-calling agents. The implicit model without an egress allowlist is deny-by-default at no domains and allow-by-default everywhere else. With an allowlist, the default inverts: the agent can reach the listed destinations and nothing else, regardless of what code or instructions it is executing.
Audit: Every action the agent takes, including actions that were blocked by enforcement, is recorded in a complete, ordered, tamper-evident log. The log is the evidence layer. It is what makes governance verifiable rather than aspirational.
These four properties correspond directly to the four questions a security reviewer asks when evaluating an agent deployment:
- Can this agent affect systems it is not supposed to touch? (Isolation)
- Can this agent consume unbounded resources? (Containment)
- Can this agent communicate with arbitrary external systems? (Access control)
- Can we reconstruct what this agent did after the fact? (Audit)
Where Policy Lives in This Framework
Policy is the specification layer. It defines what is authorized: which tools the agent may call, which egress destinations are allowed, what resource quotas apply, and what the agent is allowed to access in its sandbox. The enforcement mechanisms read and apply policy at runtime. The audit log records both the policy that was active at run start and every enforcement decision made against that policy.
Policy needs to be declarative and versioned. Declarative means that what the agent is allowed to do is stated explicitly in a configuration file, not scattered across code. Versioned means that you can determine which policy was active at any given time, which matters for audit review: if a run occurred six months ago and you need to understand what was authorized, you need the policy as it existed at that time, not the current version.
A minimal policy declaration for a governed agent looks like this:
# policy.yaml agent_id: order-processor-v2 version: 3 # isolation sandbox: filesystem: read-only env_vars: [ORDER_API_KEY, LOG_LEVEL] # containment quotas: cpu_millicores: 500 memory_mb: 256 llm_tokens: 20000 network_bytes: 2097152 # access control egress: allow: - api.orders.internal - inventory.internal # audit (always on, config only for retention) audit: retention_days: 90
Each section maps to one of the four governance properties. The policy file is the specification; the runtime reads it and enforces it at every checkpoint.
The Enforcement Model: Where Checks Happen
Governance checks must happen at the action boundary, not inside the agent's reasoning loop. This is the structural principle that separates runtime governance from application-level guardrails.
Application-level guardrails run inside the agent's trust boundary: they are part of the agent code, subject to the same vulnerabilities as the agent, and can be bypassed if the agent's reasoning is manipulated. Runtime governance runs outside the agent's trust boundary: the enforcement layer intercepts action attempts before they are dispatched, regardless of why the agent decided to attempt them.
The practical enforcement points are:
- Tool call dispatch: Before any tool call executes, check that the tool is in the authorized set, check quota headroom, and record the call in the audit log.
- Network egress: Before any network connection is established, check the destination against the egress allowlist. Block and log if not allowed.
- Resource consumption: After each operation, update the running totals for all quota dimensions. Check against limits before dispatching the next operation.
- Run termination: When the run ends, seal the audit log with the termination state and a hash that proves the record has not been modified.
Nothing about this enforcement model requires access to the agent's internal state or reasoning. It operates entirely on observable actions: what the agent tried to do, whether it was allowed, and what it actually did.
What Governance Does Not Cover
It is important to be precise about scope. Runtime governance enforces boundaries on actions within a defined permission surface. It does not:
- Prevent an agent from doing something harmful within its authorized surface. An agent with legitimate access to a write-enabled database can corrupt that database while staying inside every quota and allowlist check. The audit log will show exactly what happened, but governance did not prevent it.
- Evaluate the quality or appropriateness of the agent's reasoning. An agent that makes poor decisions within its authorized scope will make those decisions without triggering any governance control.
- Replace access control in the systems the agent calls. If the agent has a credential that grants write access to a resource, and the policy allows the agent to use that credential, governance will not second-guess that permission grant. Least-privilege credential design and governance are complementary, not substitutes.
These boundaries are not failures of the framework. They are the natural limits of what can be enforced at the runtime layer without making the governance system so opinionated that it becomes an obstacle to legitimate agent work. The goal is a layer that makes enforcement uniform and accountability complete. The layer above it, application design and access control design, determines what is authorized in the first place.
Building vs Buying the Enforcement Layer
Every platform team building agents in regulated environments needs these four properties. The question is whether to implement them in-house or use a dedicated runtime that provides them out of the box.
The in-house path is not unreasonable for teams with deep systems programming expertise. The enforcement layer for a single-tenant deployment with a small number of agents is a few hundred lines of code around the tool dispatch loop, cgroup configuration for CPU and memory, network namespace configuration for egress, and a simple append-only log. The complexity grows quickly as you add multi-tenant isolation, policy versioning, audit log integrity proofs, and support for multiple agent frameworks.
The case for a dedicated runtime is straightforward: the governance infrastructure has no competitive differentiation. It is the same for every team deploying agents in a regulated environment. The time spent building and maintaining it is time not spent on the agent capabilities that are actually specific to your use case. That trade-off is particularly clear at early stages, when every engineering week has a direct opportunity cost.
Either way, the framework is the same. Isolation, containment, access control, audit. Policy as the specification layer. Enforcement at the action boundary. An immutable record of everything that happened. Those four properties are not optional extras in a governed agent deployment. They are the definition of what "governed" means.