Back to Blog
Isolation Theory

Lessons from OS Security for Agentic AI Sandbox Isolation

· 13 min read

By Marcus Pellegrino

Before we started building Runta, I spent several years on runtime security for cloud workloads. The patterns that keep container workloads safe are not new inventions: they are adaptations of OS security research that goes back decades. Process isolation, capability-based access control, mandatory access control, least-privilege privilege separation. These ideas were worked out thoroughly in the context of operating systems, then adapted for hypervisors, then for containers. AI agent runtimes are the next surface where they apply.

What is useful about this lineage is not that the solutions transfer directly. They do not, in important ways. What is useful is that the threat models are similar enough that the OS research tells you where to focus. This post goes through the specific OS security concepts that translate well to agent sandboxing, explains the adaptation, and flags the places where the AI agent context introduces new challenges that the OS analogy does not cover.

Process Isolation: Direct Mapping, Different Implementation

Process isolation in traditional OS security means that one process cannot directly access another process's memory, file descriptors, or execution state without explicit OS-mediated sharing. The enforcement mechanism is the OS memory protection hardware combined with the kernel's access control checks on resource operations.

For AI agents, the equivalent isolation boundary is between one agent run and another, and between each agent run and the host system. The threat is the same: a compromised or misbehaving execution unit should not be able to affect the execution units around it. The implementation adapts: Linux namespaces (pid, mount, network, user) provide process-level isolation boundaries that can be applied to agent runs. A mount namespace limits the agent's filesystem view to its authorized scope. A network namespace with a controlled virtual interface applies the egress policy at the OS level, not the application level.

The direct mapping here is strong. The isolation boundary for an agent run is analogous to the isolation boundary for a process, and the kernel-level mechanisms for enforcing it are the same ones that OS security research developed. The adaptation is in how those mechanisms are configured: for a web server, the filesystem scope is the document root; for an agent, the filesystem scope is the set of paths explicitly authorized in policy, which may be smaller and more specific.

Capability-Based Security: Strong Fit for Tool Access

Capability-based security is the principle that access to a resource should be granted through a specific capability token, not through ambient authority from identity. The contrast is with ACL-based security, where "user X is permitted to access resource Y" is checked at access time against a list. In a capability model, holding the capability to access Y is sufficient; no further identity check is required.

For AI agents, tool access maps well onto the capability model. When an agent is initialized, it receives a set of tool capabilities: the specific tools it is authorized to use, with the specific credentials needed to use them. Each tool is a capability. The agent can use the tools it was given. It cannot use tools it was not given, regardless of whether those tools exist in the environment. This is directly analogous to capability-based file descriptor passing in OS research: you can operate on the file descriptors you have, not the ones you wish you had.

The practical benefit: in a capability model, privilege escalation requires obtaining an additional capability. An agent cannot simply "try" a tool it was not given access to. The set of authorized tools is defined at policy compile time, before the agent runs. This is a cleaner threat model than ACL-based tool access, where the agent might attempt to use tools and be blocked at the API level, generating noise in the security log.

Mandatory Access Control: Useful for Egress Policy, with Caveats

Mandatory Access Control (MAC) in OS security refers to access control that is enforced by the system independently of the subject's intentions. SELinux and AppArmor are the main Linux implementations. The key property: even a root process cannot override MAC policy without changing the policy itself, which requires separate administrative privilege.

For agent egress control, the MAC analogy is useful but incomplete. An egress allowlist enforced at the network namespace level has MAC-like properties: the enforcement is outside the agent's trust boundary, and the agent cannot override it through any action within the sandbox. This is stronger than application-level enforcement, which the agent could theoretically circumvent through direct socket calls or syscall manipulation if the sandbox isolation is insufficient.

The caveat: MAC in OS security was designed for labeled subjects (processes with security contexts) and labeled objects (files, sockets, ports with security contexts). The granularity of the enforcement depends on the granularity of the labels. For agent egress, the domain-level allowlist provides coarse-grained control: the agent can reach allowed domains and cannot reach unlisted domains. Subdomain-level or path-level control within an allowed domain requires application-layer enforcement, which is back inside the trust boundary. For most agent deployments, domain-level allowlisting is the right balance. For agents that call a single API with a complex surface, the domain-level allowlist may need to be supplemented with application-level path filtering for the specific endpoints in scope.

Least Privilege: The Concept Translates; the Scoping Is Harder

Least privilege in OS security means giving a process exactly the permissions it needs to do its job and nothing more. This is straightforward when you are configuring a daemon with a fixed, known function. It is less straightforward when the function is determined at runtime by an LLM reasoning chain.

For a traditional process, least privilege is a static configuration: this daemon gets read access to /etc/config, write access to /var/log/daemon, and nothing else. The daemon's function is known, finite, and expressible in a permission set that can be reviewed and approved.

For an AI agent, the function at any given run may vary within the agent's authorization scope. An agent that is authorized to process orders might read from three different record types, write to two, and call two external APIs, in different combinations depending on the specific order being processed. The least-privilege principle still applies: the authorization scope in policy should be the minimum needed for the agent to complete its defined function. But defining that scope requires understanding the full range of what the agent might need, not just what it typically does.

This is the most challenging adaptation from OS security to agent security. OS processes have well-defined permission requirements that can be determined by analysis or instrumentation. Agent runs have emergent permission requirements that depend on the specific task inputs and the model's reasoning. In practice, the approach is to authorize based on the function, not the individual run: an order processing agent gets everything it could possibly need for any valid order processing workflow, with the policy constrained to exclude everything outside that function class. This is still least-privilege in principle, but the scope is bounded by the function definition rather than by the specific execution path.

Where the OS Analogy Breaks

The most significant place where OS security concepts do not transfer cleanly is in the characterization of the "subject" whose access is being controlled.

In OS security, processes are deterministic. Given the same inputs, a process will take the same actions. This property is what makes static analysis and policy derivation possible: you can analyze a program's code to determine what permissions it needs. You can instrument a test run and observe every access it makes. The permission set derived from analysis or observation is complete and reproducible.

AI agents are not deterministic in this sense. The same agent, given the same task input, may produce different action sequences on different runs because the LLM's output is probabilistic. This means that observing a hundred runs does not guarantee you have seen every possible action sequence. The permission set derived from observation may be incomplete. An agent that has never tried to access a particular resource in any observed run might try to access it on run 101 if the model's reasoning takes an unexpected path.

This undermines the analysis-based approach to least-privilege scoping. The practical response is to bound the authorization scope by the function definition (what types of operations are logically necessary for this agent's purpose) rather than by observed behavior. This is conservative: the scope may be slightly broader than what any individual run requires. But it is more reliable than deriving scope from a sample of observed runs that may not cover the model's full probability distribution over action sequences.

The Statefulness Gap

A second place where the OS analogy breaks: state accumulation within a run.

OS process isolation is largely about protecting resources at the boundary: what a process can access, not what it accumulates. Processes do not typically carry meaningful state across invocations unless that state is explicitly persisted to storage and retrieved. Each new process invocation starts from a known, small initial state.

AI agent runs accumulate state in their context window. A run that makes 40 tool calls has collected the results of all 40 calls into its working memory. That accumulated state represents a significant data access footprint that has no direct analogy in traditional process isolation. The agent has not just touched 40 resources; it is currently holding the information content of 40 resources in its active context, and it may use that information in any subsequent action.

The implication for sandboxing: the isolation model needs to address not just what the agent can access but what it can carry across tool call boundaries. An egress check that approves sending 1KB to an allowed domain on call 40 might be approving the exfiltration of data accumulated across 39 prior tool calls. The volume of data transferred in a single call is a misleading proxy for the information content being transferred. The per-run network byte quota addresses this partially: it limits the total transfer volume regardless of which call it occurs in. But for high-stakes environments, the per-run audit trail of exactly which data was accessed and what was subsequently sent outbound is the more complete picture.

The OS security literature does not have a good answer to this problem, because the problem does not arise for traditional processes in the same way. It is a new challenge specific to agents, and the runtime tooling for addressing it is still being worked out. What we can say is that the combination of complete tool call logging with per-run audit review provides the evidence layer to detect data accumulation and subsequent transfer patterns, even where the real-time enforcement cannot evaluate the full context.

Put these controls into production

Runta gives your agents sandbox isolation, resource quotas, configurable egress allowlists, and an immutable audit trail out of the box. No custom runtime engineering required.

Request Early Access Read the Docs