Enforcing per-team AI policies in LiteLLM beyond virtual keys

What native per-team controls cover, and where they stop

Platform teams running LiteLLM as a shared proxy usually scope team access with virtual keys, team settings, budgets and model allowlists. These are access controls. They answer three questions: who is calling, which models that caller may use, and how much or how fast it may call.

That is a necessary layer, and nothing in this guide replaces it. The gap is what the mechanisms do not address. A key with a valid model allowlist can still send a prompt containing data the team should never send to a model. A response can still contain content the team's policy forbids. An agent behind that key can still call a tool or MCP server that its team has no business using. And nothing in key scope says when an interaction should pause for a person to approve it.

The second problem is evidence. Budgets and allowlists show that a team was configured a certain way. They do not show that a specific policy was applied to a specific interaction and what action resulted. Auditors and compliance teams tend to ask for the second kind of proof.

Access control and policy enforcement are different jobs

It helps to keep the two concerns apart in design. Access control is identity-centred: authenticate the caller, then permit or deny use of a resource. Policy enforcement is interaction-centred: inspect a particular request, response or tool call, then choose an action such as allow, redact, block, rate limit or escalate.

NIST SP 800-204B gives a useful vocabulary. It describes an access-control framework with an authenticatable runtime identity for services and a Policy Enforcement Point (PEP) that can be deployed and controlled separately from the application. The Policy Decision Point (PDP) can be a sidecar proxy or an external service. The framework also lists logs and metrics for monitoring policy enforcement as a component, so evidence is part of the design from the start.

The same document describes attribute-based policy, where rules are written against attributes without prior knowledge of the specific users and resources governed. A team identifier fits naturally as one such attribute. Mapping LiteLLM team metadata to a policy attribute is a design choice on your side rather than a built-in behavior, so treat it as an architecture decision to verify in your deployment.

What to record for every enforcement decision

An audit conversation usually reduces to one question: which policy applied to this team's interaction, and what happened? Design the record to answer it without reconstruction. At a minimum, capture the team identifier, the policy and its version, the interaction or tool call evaluated, the action taken (allow, redact, block, rate limit or escalate), and a timestamp. For redaction, record what category was matched rather than the raw content, so the evidence does not become a second copy of the sensitive data.

For escalations, also record the approver, the outcome and the time the decision was made. NIST AI RMF guidance calls for clear roles for the humans overseeing AI systems, and the approval record is where that assignment becomes visible. The NCCoE paper also points to visibility into prompts and data input sources as support for risk determinations, so decide deliberately how much of each interaction your records retain.

Retention is an organizational decision. Set it per the obligations that apply to you, and confirm that the retention period of enforcement records matches the period of any other records an audit would compare them against.

Where Trussed fits

Trussed is one way to provide the enforcement layer described above. It enforces policy in the execution path across models, agents, tools and MCP servers, with the actions allow, redact, block, rate limit and escalate for human approval. Scope is the individual interaction and tool call, which is the level where per-team content and tool rules operate. Policy is enforced at runtime, not reviewed after the fact.

Enforcement itself generates the audit-ready evidence, mapped to HIPAA, NAIC, NYDFS, EU AI Act and NIST. Trussed deploys self-managed in the customer cloud, with configurable retention and data residency, which bears directly on the retention and data-handling decisions above. In production, it has run continuously for more than 18 months in a regulated enterprise, with 100+ models under one policy plane.

Key scope, budgets and model allowlists remain access-control concerns. The enforcement layer complements them by deciding what each team's interactions may contain and do.