Post-Deployment Monitoring for AI

Why Post-Deployment Monitoring Is a Distinct Discipline

Pre-deployment evaluation tests a model or agent against a fixed set of prompts, benchmarks, or scenarios before it goes live. Post-deployment monitoring is a different activity: it observes what the system actually does once it is operating on real inputs, with real permissions, against real data and tools.

NIST's AI Risk Management Framework reflects this distinction directly. Its "Manage" function requires organizations to monitor AI system risks on an ongoing basis after deployment, not only during design and testing. A model that passed evaluation can still behave differently in production because inputs shift, integrations change, or an agent is granted broader tool access than was tested.

Treating monitoring as a continuation of evaluation, rather than a separate runtime discipline, is one of the most common gaps enterprises encounter when trying to detect drift or policy violations after launch.

How This Differs from Traditional APM and Model Evaluation

Conventional application performance monitoring captures latency, error rates, throughput, and infrastructure health. These signals matter for AI systems too, but they do not reveal what an agent decided to do, which tools it invoked, what parameters it passed, or whether its output stayed within an authorized scope.

Monitoring AI agents at runtime requires capturing behavioral and semantic signals in addition to standard performance metrics: tool calls, decision outputs, and permission usage. Drift detection is similarly distinct from evaluation. Evaluation compares a model against a fixed test set before release. Drift detection compares live output or decision distributions against an established baseline on an ongoing basis, which depends on having set that baseline before the system entered production rather than reconstructing one after an incident.

Required Components for Production-Grade Oversight

Production-grade monitoring for AI agents generally requires five interlocking components:

• Behavioral monitoring: observes decision patterns and output characteristics over time.
• Tool-call logging: records every function invocation an agent makes, including parameters, accessed resources, and outcomes, a level of detail that is not a standard feature of conventional observability tooling.
• Drift detection: measures deviation from a pre-established behavioral baseline rather than from a static benchmark.
• Policy enforcement: introduces a decision layer that evaluates agent actions against approved rules, a materially different function from passive logging or dashboarding because it can intervene rather than only report.
• Audit trails: tie the above together into records sufficient to reconstruct what an agent did, why, and under what authorization, necessary for both internal review and regulatory response.

Agent identity and permission scope tracking underpin all five components, since monitoring signals are only actionable when they can be correlated to a specific agent and its authorized boundaries.

Governance and Regulatory Context

Several frameworks now treat post-deployment oversight as a required governance activity rather than an optional enhancement:

• The EU AI Act establishes post-market monitoring obligations for providers of high-risk AI systems, requiring monitoring plans proportionate to the system's risks.
• ISO/IEC 42001, the first international standard for AI management systems, requires documented processes for monitoring AI system performance throughout its operational lifecycle as part of a certifiable management system.
• NIST's Generative AI Profile (AI 600-1) extends the AI RMF to address risks specific to generative systems, including behavior that only emerges after deployment.
• OWASP's LLM risk guidance identifies excessive agency and insecure output handling as runtime concerns, and MITRE ATLAS catalogs adversarial tactics observed against deployed AI/ML systems specifically, as distinct from pre-deployment benchmarks.

Evaluation Criteria for a Monitoring Approach

When assessing a monitoring approach for production AI agents, consider the following questions:

• Does it capture tool-call-level detail, including parameters and outcomes, rather than only aggregate performance metrics?
• How is the behavioral baseline established and kept current as the system evolves?
• Can policy violations be prevented at runtime, or only reported after the fact?
• Are agent identities and permission scopes tracked and linked to corresponding log entries?
• What retention, immutability, and export capabilities exist to support audit and regulatory review?