When we started talking with platform teams at financial services firms and healthcare organizations about what they needed before they could put agents into production, we expected the conversation to focus on model safety: what the LLM might generate, how to prevent inappropriate outputs, whether the model provider's content policies were sufficient. Those questions came up, but they were not the first or hardest questions. The first questions were about access control, audit trails, and who bears responsibility when an agent does something that triggers a review.
The gap we kept running into: the tools that LLM providers and most agent frameworks offer for safety are prompt-level controls. They operate inside the agent's trust boundary. For teams in regulated environments, the security review starts at the boundary between the agent's execution environment and the rest of the system. Prompt filters cannot speak to that boundary at all.
What Security Reviews Actually Ask
A security review for a new system in a regulated environment follows a relatively consistent structure, whether it is conducted by an internal InfoSec team, an external auditor, or a regulatory examiner. The questions map to a few core concerns:
Access scope: What data can this system reach, and what external systems can it communicate with? Can you demonstrate that its access is limited to what it needs for its function?
Least privilege: Does the system have the minimum access required, or does it have broader access that could be misused?
Audit accountability: Can you produce a complete record of what this system did, when, and using what credentials?
Incident response: If something goes wrong, what is the evidence trail? Can you determine the scope of impact?
None of these questions are answered by a model provider's content policy or a prompt filter. A content policy prevents the model from generating harmful text. A security review is asking about network access, credential use, and data access patterns. Those are infrastructure concerns, not model behavior concerns.
The Trust Boundary Problem
The fundamental architectural issue is where prompt-level controls sit relative to the trust boundary. An AI agent that has been given credentials and can make API calls operates on both sides of a trust boundary that matters to security reviewers:
- Inside the trust boundary: the LLM call, the prompt construction, the reasoning that decides what to do next. This is where prompt filters operate.
- Outside the trust boundary: the actual API calls, the network connections, the data reads and writes that produce real-world effects.
Prompt filters and application-level guardrails run inside the trust boundary. They are subject to the same vulnerabilities as the agent: if the agent's reasoning can be manipulated (through prompt injection, adversarial inputs, or edge cases in the model's behavior), the guardrails can be manipulated along with it. The security reviewer's question is not "can the model be tricked?" but "if the model is tricked, what can it actually do?"
Runtime controls operate outside the trust boundary. They intercept actions before they produce effects, independent of the agent's internal reasoning state. An egress allowlist does not care whether the agent was manipulated into trying to reach an unauthorized domain. It blocks the attempt regardless. A quota check does not reason about whether the agent has a good reason to make 1,000 API calls. It counts them and terminates the run if the limit is reached.
A Concrete Example: Payment Processing Workflow
Consider a plausible scenario: a small fintech team building an agent to automate payment exception handling. The agent processes incoming payment exceptions, retrieves account data, performs classification and routing logic, and updates records in the payment platform. In a regulated environment, that workflow involves access to account data, potential writes to payment records, and outbound calls to partner APIs for verification.
The security review for this agent will ask: what account data can the agent read, and can it write to records beyond the exceptions it is processing? What external APIs can it call? Is there a complete record of every action it takes, including failed or blocked attempts? If an exception is misprocessed, can you trace the agent's decision chain?
Prompt filters address none of these questions. An egress allowlist limits outbound calls to the explicitly authorized partner verification APIs. Quota enforcement prevents the agent from reading or writing beyond the expected volume per run. An audit log produces the complete action record the review requires. Sandbox isolation prevents the agent from accessing account data outside its authorized scope even if its reasoning would lead it there.
We are not claiming that prompt filters are useless in this scenario. Preventing the agent from generating harmful or inappropriate text as part of its outputs has real value. But the security review for a payment processing workflow is looking at access controls and audit trails, not at the quality of the agent's written reasoning. These are two different layers, and both need to be addressed.
The Compliance Trail That Prompt Filters Cannot Generate
In healthcare environments, a common regulatory concern is access to patient-adjacent data: who touched it, when, and what did they do with it? This is the minimum information needed to respond to a data access inquiry, whether from a patient exercising their rights or from a regulatory examiner.
An agent that retrieves patient-adjacent data as part of an automated workflow creates an access event. That event needs to be in the audit trail with enough detail to reconstruct what was accessed, by what process, under what authorization, and what happened to the data afterwards. A prompt filter generates no record of this. The LLM call that processed the data may produce a record with the model provider, but that record is in the provider's custody, not yours, and may not include the specific data access details your compliance trail requires.
The audit record for a governed agent run captures the tool call that retrieved the data (with the record identifier), the egress decisions for any outbound calls that involved the data, and the termination state of the run. That record is in your control, stored with your retention policies, and contains the fields that satisfy the compliance question. The record is complete because it is captured at the runtime layer, which intercepts every action regardless of what the agent's reasoning produced.
What "Defense in Depth" Means for Agent Deployments
The appropriate framing is not "runtime controls instead of prompt filters" but "runtime controls in addition to whatever model-level controls you are using." Defense in depth means multiple independent layers, each addressing the threats that the other layers cannot.
Model-level controls (content policies, moderation classifiers, system prompt instructions) address the quality and appropriateness of the agent's reasoning and output. They are the right layer for preventing the agent from generating harmful text, maintaining task focus, and avoiding policy-violating outputs.
Runtime controls (sandbox isolation, quota enforcement, egress allowlisting, audit logging) address what the agent can actually do in the environment it is running in. They are the right layer for the questions that security reviews ask: access scope, least privilege, accountability, incident response.
Platform teams that deploy only model-level controls have addressed the model safety dimension and left the infrastructure security dimension to chance. Security reviews in regulated environments will find that gap, and it is not addressable with additional prompt engineering. The controls at those two layers are not substitutes for each other because they operate on different things: one on the agent's outputs, the other on the agent's effects.
What changed recently to make this tension visible is the shift from agents that observe and suggest to agents that act. An agent that reads data and summarizes it is a read-only observer. An agent with write credentials and the ability to make outbound network calls has a real-world surface that is exactly what regulated environments have always required security controls for. The stack of controls needed did not change. What changed is that agents now need it too.