LiteLLM vs Kong vs Azure API Management: Extensibility for Policy Enforcement on AI Traffic

Why the categories matter more than the feature lists

Platform teams usually ask which gateway has the most AI features. A more durable question is how each one lets you add policy that the vendor did not ship. Custom policy is where the effort, the skills requirement, and the operational burden accumulate.

LiteLLM is a proxy server that exposes LLM APIs in OpenAI (or native) format. Its extension model works inside that proxy: hooks, callbacks, and configuration. Kong is a general-purpose API gateway that adds AI capabilities through plugins, and its plugin model lets you write custom logic in several languages. Azure API Management is a managed cloud service. Its policies are declarative XML with expressions, and it reaches other Azure services when the logic outgrows the policy language.

These are not interchangeable choices. A team already running Kong for all API traffic and a team that needs a single OpenAI-compatible front door for many model providers are solving different problems, and the extension model follows from that starting point.

Extension mechanics side by side

The table summarizes the extension model for each product at the level a platform team needs to plan effort. Stage coverage and built-in policy types should be confirmed against current product documentation for the version you deploy.

• Category: LiteLLM: open-source LLM proxy. Kong: general-purpose API gateway with AI plugins. Azure API Management: managed cloud API management service.
• Extension model: LiteLLM: proxy hooks, callbacks, and configuration. Kong: plugins. Azure API Management: XML policy expressions plus integrations with other Azure services.
• Custom code language: LiteLLM: code that runs in the proxy alongside configuration. Kong: Lua, Go, JavaScript, or Python. Azure API Management: policy expressions, with heavier logic moved to an external Azure service.
• Who runs the runtime: LiteLLM: your team, as a self-operated proxy. Kong: depends on how you deploy it. Azure API Management: the cloud service is managed, but your team owns the policy content.
• Natural fit: LiteLLM: unified access to many models with policy near the proxy. Kong: one gateway standard across API and AI traffic. Azure API Management: teams standardized on Azure.

Built-in versus custom policy: how to judge the gap

For each policy type you need, decide whether the gateway configures it or you must build it. The types in the brief are content filtering, redaction, authorization, rate limiting, and approval steps.

Rate limiting is the most standardized. NIST's guidance on API protection lists enforcing limits on API and resource usage as a recommendation, and advises robust threshold-setting such as token bucket, leaky bucket, or adaptive limits based on user behavior or service tier, rather than arbitrary numbers. For LLM traffic, ask whether the native limiter counts tokens or only requests, and whether you can key limits by user, team, or model. If not, that limiter becomes custom code.

Schema validation is another useful test. NIST notes that machine-readable API documentation lets runtime tooling validate request and response schema and syntax. Check whether the gateway validates LLM payloads that way or leaves it to you.

Content filtering and redaction are where custom effort tends to concentrate. Detection quality usually depends on an external classifier or service, so the extension point mostly becomes the place that calls it. Redaction on streamed responses is harder than on complete responses, because content arrives in fragments. Approval steps are harder still: a synchronous request path has no natural place to wait for a human, so approval typically requires an external workflow and a way to resume or reject the request.

Authoring, testing, deployment, and ownership

Each extension model implies a different delivery pipeline. Plugin code in Kong or hook code in LiteLLM is software: it needs source control, unit tests, a staging environment, and a release process. XML policies in Azure API Management are configuration, but they still need versioning and testing, and expressions that grow complex become hard to review. Treat all three as code under your normal software development governance. NIST's generative AI profile advises connecting new AI policies and processes to existing IT, software development, data, and risk governance.

Operational ownership differs by category. A self-managed proxy or gateway makes your team responsible for patching. As an example, NVD lists CVE-2026-42271, a command injection vulnerability in LiteLLM affecting versions from 1.74.2 up to but not including 1.83.7, with the required action to apply vendor mitigations. That is a reminder about any self-operated gateway, not a judgment of one product: the gateway is part of your attack surface, and patch cadence is a standing duty. A managed service moves runtime patching to the provider, while policy logic, testing, and change control stay with you.

Policy format also matters for assurance. NIST's OSCAL work describes policy controls implemented as rules and tested through rule engines or checks based on evidence. Policies kept in machine-readable, testable form are easier to verify than logic scattered across ad hoc scripts.

Evidence and where gateway-level policy ends

A gateway can only enforce policy on traffic that passes through it. That is a general property of the architecture, and it defines the limit. If an agent calls a tool directly, or reaches an MCP server without routing through the gateway, no gateway policy sees the call. Even when traffic is routed through the gateway, a request-level view may not carry the context needed to decide whether a specific tool call is in scope for a specific user or task.

Evidence is the second limit. Define the decision records you need before choosing an extension model: what was evaluated, the decision (allow, redact, block), the policy version, the identity, and any approver. Then confirm that the extension point can emit those records in a form you can retain and export. Evidence-based control testing is an established compliance pattern, and a gateway's operational log is not automatically an audit record of enforcement decisions.

NIST's zero trust guidance for cloud environments expects granular, application-level policy. For AI, that maps to per-user, per-tool, and per-model authorization. When those requirements apply, or when a regulated enterprise must show what its AI was allowed to do and what it did, enforcement in the execution path is a separate layer from the gateway's extension points. Trussed enforces policy there across models, agents, tools, and MCP servers (allow, redact, block, rate limit, and escalate for human approval) and produces audit-ready evidence from that enforcement. It applies to individual interactions and tool calls, and it can run self-managed in the customer cloud. It complements a gateway rather than replacing one.