Continuous Monitoring and Review of AI Risks Under ISO/IEC 23894 Using Runtime Evidence

Turning monitoring and review into operational requirements

ISO/IEC 23894:2023, titled Guidance on risk management for AI, describes processes for implementing and integrating AI risk management. It is guidance. ISO/IEC 42001 is the standard that specifies requirements for an AI management system, so 23894 is best positioned as support for that system rather than as a certifiable requirement set. As of November 2026, the ISO catalogue lists ISO/IEC 23894:2023 with no later revision.

Monitoring and review is where documented risk decisions meet production reality. The mappings in this guide from review activities to runtime evidence are design inferences, not clause text. Consult the standard itself before citing a specific requirement in an audit response.

The NIST AI RMF is a useful companion for making this concrete. It states that risk management should be continuous, timely and performed throughout the AI system lifecycle. Govern 1.5 calls for ongoing monitoring and periodic review of the risk management process and its outcomes, with clearly defined roles and a determined review frequency. The NIST Playbook adds an action to establish the frequency and detail of monitoring, auditing and review.

Three operational requirements follow. First, you need evidence that each risk treatment operated, not only that it was approved. Second, you need a defined cadence and detail level for review. Third, review must cover the risk management process itself, not only the behavior of individual systems.

What runtime evidence looks like at the interaction and tool-call level

Periodic assessments, manual attestations and after-the-fact log reviews share a weakness. They describe intent or sample behavior, and they rarely show what a control did on a given interaction. Runtime evidence is different when policy is enforced in the execution path, because the decision and the action are recorded as part of the event itself.

Useful categories include policy decisions, enforcement actions (allow, redact, block, rate limit), human approval escalations, redaction events and rate limit events. Each category substantiates that a specific risk treatment operated. A redaction event, for example, supports a treatment for sensitive data exposure. A human approval record supports a treatment that requires oversight for consequential actions.

What makes evidence usable for audit is a consistent record structure. The following field list is a design recommendation, not a standard-defined schema.

• Timestamp and a unique interaction or tool-call identifier
• Policy identifier and version that was evaluated
• Decision and enforcement action taken, with the reason
• Identity of the requesting user, application or agent, and the target model or tool
• Escalation record: approver, decision and time to decision
• Retention and residency attributes for the evidence itself

Assigning review responsibilities and setting cadence

Govern 1.5 is explicit that roles and responsibilities for monitoring and periodic review should be clearly defined. Evidence without an owner becomes an archive. Name a risk owner for each register entry, a reviewer who is independent of the team operating the system, and an escalation path when evidence contradicts the recorded risk rating.

Cadence should reflect risk. A tool call that can change a customer record warrants more frequent, more detailed review than a low-impact summarization use case. Write the frequency and detail into the review procedure so internal audit can test against it, and revisit them when trigger events show the original choice was wrong.

Also decide how you will test your own metrics. The NIST Measure function includes assessing whether metrics are effective at identifying and measuring emergent risks. A dashboard of block counts that never changes a decision is a signal to revise the metric, not to reduce review.

Auditors and regulators typically ask what a system was permitted to do and what it did. Evidence generated by enforcement can answer that for individual interactions and tool calls. Trussed enforces policy in the execution path and produces audit-ready evidence from that enforcement, mapped to frameworks including NIST and the EU AI Act. The review process around that evidence remains the organization's responsibility.

Where runtime evidence helps and where other processes are needed

Runtime enforcement evidence covers one part of monitoring and review. NIST AI 800-4, published in March 2026, describes monitoring as necessary to validate reliable operation in real-world scenarios, track unforeseen outputs and drift, and identify unexpected consequences in new or changing contexts. Policy-decision evidence alone does not establish those things. It also does not establish model provenance, transaction-level governance or end-to-end tracing of agentic workflows, so plan other mechanisms for those needs.

• Did the treatment operate on this interaction?: Runtime evidence: policy decision and enforcement action per interaction or tool call. Other process: periodic sampling to confirm policy coverage.
• Was human oversight exercised?: Runtime evidence: escalation and approval records. Other process: review of whether approvers are qualified and approval criteria are sound.
• Is the system drifting or producing unforeseen outputs?: Runtime evidence: partial at best. Other process: output quality evaluation and outcome monitoring against defined metrics.
• Are risk ratings still correct?: Runtime evidence: input through trends and trigger events. Other process: risk owner judgment, impact assessment and register governance.
• Is the review process itself effective?: Runtime evidence: not applicable. Other process: internal audit, management review and metric effectiveness testing.