Why LLM Governance Programs Fail and How to Close the Gap Between Written Policy and Runtime Enforcement

Governance failure is an enforcement and evidence problem

Most regulated enterprises already have an LLM policy, an approval committee and an acceptable-use standard. Programs still stall because those artifacts describe intent. They do not constrain what a model, an agent or a coding assistant does when a prompt arrives or a tool is called.

The gap shows up in predictable ways. Teams use unapproved AI tools outside the approved path. Controls differ from one model to the next. Nobody is clearly accountable for what an agent may do. Review happens after the fact, when the action has already occurred. When an auditor asks what the AI was allowed to do and what it did, the answer is assembled from policy documents, meeting minutes and scattered logs.

There is also a measurement problem. NIST's AI Risk Management Framework notes that AI risks which are not well defined or adequately understood are difficult to measure, and that failure modes of emergent properties are harder to predict. A written policy cannot anticipate every runtime behavior. That is an argument for controls that act on actual behavior and for evidence that records it, not for longer documents.

Six structural gaps, each with a fix and a check

Use this list as a self-assessment. For each gap, the check is something you can run, not something you can attest to in a document.

• 1. Policy that cannot be tested: Fix: rewrite each policy statement so a test could show a violation, and pair it with a verification method. Check: pick one statement and show the test that would fail if the policy were broken.
• 2. Review after the fact: Fix: enforce policy in the execution path so a prohibited action is stopped before it completes. Check: attempt a prohibited action in a test and confirm it did not execute.
• 3. Uneven coverage and shadow AI: Fix: apply one policy across models, agents, tools and coding assistants, and route usage through that path. Check: run the same test against each model and tool and compare outcomes.
• 4. Unbounded agency in tool calls: Fix: inventory where systems can act, classify high-impact actions, and require approval for them. Check: trigger a high-impact action and confirm approval was required and recorded.
• 5. Unclear ownership: Fix: name owners for policy content, agent and tool permissions, and approval decisions. Check: for any exception in the last quarter, identify who approved it.
• 6. Fragmented evidence: Fix: generate evidence where enforcement happens rather than reconstructing it later. Check: choose one interaction and reconstruct what was permitted and what occurred without manual assembly.

What enforcement in the execution path means

Enforcing a policy in the execution path means the decision is made before the interaction or tool call completes, by a component the model cannot bypass. The possible outcomes are limited and worth stating explicitly: allow, redact, block, rate limit, or escalate for human approval. Each should be a defined result of a policy rule, not a judgment made later in a review queue.

Tool calls deserve the most attention. OWASP describes LLM-based systems as often being given agency, the ability to interface with other systems and take actions in response to a prompt. It names excessive autonomy, where high-impact actions are not independently verified and approved, as a cause of excessive agency. Its Cornucopia card on the topic describes the exploitable condition as the AI being allowed to act without human-in-the-loop approval. A human escalation path for high-impact actions is therefore a core control, not an optional feature.

Enforcement has limits that should be stated plainly. OWASP's prompt injection guidance notes that screening does not stop an injection it fails to detect and does not cover content arriving through other channels. Input screening should be one layer, paired with action-level limits so that a missed injection cannot produce an unapproved high-impact action. Scope also matters. Enforcement at the level of individual interactions and tool calls does not by itself give you a view of a whole multi-step workflow, so be clear about what your controls do and do not cover.

Permissions should be narrow and time-bound. NIST's design tenets for agent identity management include short-lived, dynamic and revocable authorization. Applying that idea means an agent's credentials carry a limited scope and lifetime, and access can be withdrawn without redesigning the system. NIST and its Center for AI Standards and Innovation also recognize that monitoring deployed AI systems is challenging, so plan for incomplete visibility rather than assuming it.

Ownership and evidence: what an auditor can actually use

Ownership fails when responsibility is spread across a committee. Assign accountable owners for three things: the policy rules themselves, the permissions granted to agents and tools, and the approval decisions for escalated actions. An escalation path with no named approver and no record of the decision is a gap even if the control technically exists.

Evidence is where many programs are weakest. Policy PDFs, committee minutes and training records show that a program exists. They do not show what a specific model or agent was permitted to do, or what it did, in a specific interaction. Log exports assembled after an incident have a similar weakness: they are partial, inconsistent across systems, and disconnected from the policy rule that applied.

Useful evidence is produced as a by-product of enforcement. For each interaction or tool call, it should show which policy applied, what decision was made (allow, redact, block, rate limit or escalate), and, for escalations, who approved and when. It should be retained for a period you set, stored where your residency requirements allow, and be durable enough to survive staff and tooling changes.

Map that evidence to the frameworks that apply to you, such as NIST AI RMF, the EU AI Act, HIPAA, NAIC and NYDFS requirements. Keep the mapping versioned and review it when texts change. Where the NIST AI RMF acknowledges that some risks are hard to measure, document the residual uncertainty instead of claiming full coverage.

One example of enforcement in the execution path

Trussed is one example of this approach. It enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants, with outcomes of allow, redact, block, rate limit or escalate for human approval. Scope is individual interactions and tool calls. Enforcement is at runtime, not reviewed after the fact, and the audit-ready evidence it produces is mapped to HIPAA, NAIC, NYDFS, the EU AI Act and NIST.

It can be self-managed in the customer cloud, with configurable retention and data residency. In production, it has run continuously for 18+ months in a regulated enterprise, holds 100+ models under one policy plane, and has been delivered to customers in insurance, pharma, healthcare and education.

The approach is only useful if it closes the gaps above in your environment. Apply the same checks to it as you would to any other: run the prohibited-action test, compare behavior across models, and reconstruct a single interaction from the evidence.