Implementation Guide

    How to Replace Azure OpenAI API Keys with Managed Identity and Enforce Policy on Every Call

    To replace Azure OpenAI API keys with managed identity, route all workloads through a proxy that holds the only managed identity with inference access. Grant that identity a least-privilege role on the resource, migrate each consumer to the proxy, then disable local key authentication on the resource. Managed identity establishes who is calling Azure OpenAI. The proxy decides what each caller may send and receive on each request.

    1. 1

      Proxy identity

      A system-assigned or user-assigned managed identity on the proxy workload. A user-assigned identity can simplify replacement or scale-out of proxy instances.

    2. 2

      Role assignment

      A least-privilege data-plane role that permits inference calls only, scoped to the specific Azure OpenAI resource rather than a resource group or subscription.

    3. 3

      Resource setting

      Local key authentication disabled on the resource, so key-based requests are rejected.

    4. 4

      Caller identity

      Each inbound caller authenticates to the proxy with its own workload identity, such as short-lived certificates or signed tokens, so decisions and records are attributed per workload.

    5. 5

      Policy path

      The proxy enforces on requests and responses. Where decisions depend on end-user attributes, use external authorization rather than transport identity alone.

    6. 6

      1. Discover key consumers

      Inventory code repositories, pipelines, secret stores and configuration that hold keys, and use resource diagnostics to see which callers still authenticate with them.

    7. 7

      2. Deploy the proxy

      Stand it up with its managed identity and the scoped role, initially in a permissive mode that allows and records traffic.

    8. 8

      3. Give callers their own identities

      Issue each workload an identity the proxy can verify, so attribution does not depend on a shared secret.

    9. 9

      4. Cut over by workload

      Change each client's endpoint and credential handling to use the proxy. Migrate low-risk workloads first and keep keys active meanwhile.

    10. 10

      5. Turn on enforcement

      Move from record-only to enforcement per workload, once observed traffic shows policies behave as intended.

    11. 11

      6. Disable local authentication

      After diagnostics show no key-based calls over an agreed period, disable key access on the resource and remove stored keys.

    Validation checks that confirm keys are retired

    • Local authentication shows as disabled on every Azure OpenAI resource in every environment.
    • A test call using a previously issued key is rejected.
    • Azure Monitor and diagnostic logs show no key-authenticated requests across a full business cycle.
    • No route to the model endpoint exists except through the proxy's identity, including from build agents and developer machines.
    • A test request that violates policy is blocked or redacted, and the decision appears in the proxy's records.
    • Repository and pipeline scans find no remaining Azure OpenAI keys.

    Keyless authentication and runtime policy are separate controls

    Long-lived shared keys fail in three ways. They are copied into code, pipelines and configuration, so rotation is slow. They do not identify which workload made a call. They also bypass any central policy check, because anyone holding the key can reach the model directly.

    Moving to Microsoft Entra managed identity fixes the first problem and part of the second. Credentials are issued to an Azure identity rather than stored in a file, and access is granted through role assignments that can be reviewed. It does not decide what a given workload is allowed to send to a model or what it may receive back. A workload with a valid identity and a broad role can still submit sensitive data or request output it should never see.

    This maps to a familiar pattern in zero trust architecture. NIST SP 800-207 describes access to a resource as granted through a policy decision point and a policy enforcement point. Authentication tells you the subject. The enforcement point applies policy to the specific request. Treat the two as separate controls and deploy both.

    Failure modes and rollback

    Residual key use is the most common failure. A forgotten batch job or a copied key in a notebook keeps working until local authentication is disabled, so discovery has to continue after cutover rather than end at the inventory. Disabling local authentication is the control that makes the migration real, because it turns any missed consumer into a visible failure instead of a silent bypass.

    Client token handling is the second risk. Clients that call the proxy should obtain and refresh tokens through their SDK's credential mechanism rather than caching tokens in configuration or environment variables. A long-lived token pasted into a pipeline reproduces the key problem under a different name.

    Identity scoping is the third. If the proxy's identity is granted a role at subscription scope, or several unrelated resources share one proxy identity, a compromise reaches further than intended. Scope the role to the resource and separate identities by environment.

    Plan rollback before cutover. If the proxy or its identity fails, the fallback is a time-boxed, approved re-enablement of key access, followed by rotation of any previously distributed keys. Run the proxy with redundancy so that fallback is rare, and rehearse it. Every use of the fallback should itself be recorded.

    What the enforcement layer should record

    Removing keys improves attribution only if the proxy keeps its own record of each interaction. For every request, capture the authenticated caller identity, the timestamp, the policy that applied, the decision, and the action taken: allow, redact, block, rate limit or escalate for human approval. Where escalation occurs, record who approved or rejected it. Retain the records in a protected store with a defined retention period, because they are the evidence an auditor or regulator will ask for. The specific fields a framework expects vary, so confirm the mapping with your compliance team.

    Trussed enforces policy in the execution path, so decisions are enforced at runtime, not reviewed after the fact. It produces audit-ready evidence from that enforcement, mapped to HIPAA, NAIC, NYDFS, the EU AI Act and NIST, and deploys self-managed in the customer cloud with configurable retention and data residency. The pattern in this guide does not depend on any one product.

    Keyless request path

    Workload

    Holds no Azure OpenAI key and presents its own identity to the proxy.

    Policy enforcement proxy

    Applies policy to each request and response: allow, redact, block, rate limit or escalate.

    Managed identity

    Held by the proxy and used to obtain tokens for the Azure OpenAI resource.

    Azure OpenAI resource

    Local key authentication disabled, so Microsoft Entra ID is the only way in.

    Plan your keyless migration

    Talk through your key inventory, proxy placement and evidence requirements with a Trussed AI expert.

    Talk to an Expert
    Next step

    How does your AI governance program compare?

    See where your program has gaps in less than 2 minutes.