Setting AI risk criteria and likelihood-impact scales under ISO/IEC 23894

Why criteria come before assessment

ISO/IEC 23894:2023 gives guidance on how organizations that develop, produce, deploy or use AI products, systems and services can manage risk specifically related to AI. It aims to help organizations integrate risk management into AI-related activities and functions. As of November 2026, it is a voluntary guidance standard, not a law or regulation.

This guide follows the order of the risk management process: scope, context and criteria, then assessment and treatment. Criteria come first because assessment results are only comparable when every team scores against the same definitions. ISO 31000 defines risk as the effect of uncertainty on objectives, so criteria have to be anchored to objectives and obligations, not to a generic score.

Many teams reuse a generic enterprise risk matrix. That tends to fail for AI because the matrix has no way to express output variability, dependence on data, or the autonomy of a system that calls tools. Two teams then score the same system differently, and an acceptance decision cannot be traced to a defined threshold.

Define context and risk appetite inputs

Start by recording what the scales must serve: the objectives the AI system supports, the legal, regulatory and sector requirements that apply, and the organization's stated appetite. The NIST AI RMF defines risk tolerance as the readiness of an organization or AI actor to bear risk to achieve its objectives, and notes it can be influenced by legal or regulatory requirements. It does not prescribe tolerance, so the organization must own that decision.

NIST also describes tolerance and acceptable risk as contextual and use-case specific. Follow existing regulations and guidelines for risk criteria where your sector or profession already has them, and define a reasonable tolerance only where none exist.

Produce a short criteria record with the following content: the approving owner, the scope of systems covered, the risk sources considered, and a version identifier. The NIST Playbook lists financial, operational, safety and wellbeing, business, reputational and model risk as sources to consider. Use several of them rather than a single generic impact dimension.

Design bounded likelihood and impact scales

A usable scale has a fixed number of levels, and each level has a descriptor that two reviewers would apply the same way. The NIST Playbook describes a common approach of combining estimated impact and likelihood into a risk score and placing it on a scale, which may be qualitative, such as red-amber-green. The method is standard. The descriptors are where AI-specific work happens.

For likelihood, tie levels to observable evidence rather than opinion. Bound each level by a defined population (interactions, tool calls, a time window) and a defined undesired outcome. Account for variability: the same input can produce different outputs, so a single clean test run does not justify a low rating.

For impact, define descriptors per risk source and per scope. NIST's Generative AI Profile notes that risks vary by lifecycle stage and by scope (model, system, application, ecosystem). It also lists tiering factors worth building in: information integrity impacts, dependencies between the AI system and other IT or data systems, harm to fundamental rights or public safety, and objectionable, discriminatory, invalid or untruthful output.

The example below is illustrative. Calibrate every descriptor to your own context.

• Likelihood 1 (Remote): No undesired outcome observed across a defined evaluation population, and no known mechanism that produces it.
• Likelihood 3 (Possible): Undesired outcome observed in evaluation or production at a rate you define, for example per stated number of interactions.
• Likelihood 5 (Expected): Undesired outcome occurs routinely under normal use, or the system can take the action without a control stopping it.
• Impact 1 (Minor): Incorrect output is caught before use and affects no regulated decision or external party.
• Impact 3 (Significant): Output feeds a business process and may require correction, notification or rework.
• Impact 5 (Severe): Harm to individuals, exposure of regulated data, or an irreversible action taken through a tool call.

Set acceptance and escalation thresholds, then calibrate

A threshold is a decision, so write it as one. Map each band of the combined score to a defined outcome: accept, accept with monitoring, treat, escalate to a named approver, or decline. Resources are finite, so NIST suggests directing greater oversight and mitigation to systems with lower tolerance. Escalation rules should say who decides, within what timeframe, and what must be recorded.

The Generative AI Profile also suggests minimum performance or assurance thresholds reviewed as part of deployment go/no-go approval. That is a useful precedent for treating thresholds as gates. Extending a gate to individual runtime decisions is a design choice, covered in the next section.

Calibrate by scoring a small set of real or representative systems with several reviewers, then compare results and resolve disagreements by tightening descriptors. Recalibrate on a schedule and when new information arrives. Keep each version, so you can show which criteria applied to a given decision.

• Reusing a generic matrix without AI-specific descriptors.
• Levels defined by adjectives only, with no population, window or outcome.
• Metrics that reflect factors unrelated to the underlying impact, which NIST cautions against.
• Scoring the model alone when impact depends on the tools and data it reaches.
• Thresholds with no named approver or no linked action.
• Editing scales without versioning, which breaks audit traceability.

Translate thresholds into runtime policy (design guidance)

A threshold on paper does not change what a system does. To make it enforceable, express each threshold band as a policy condition evaluated where the interaction happens, and bind it to an action. A practical mapping is: allow when the interaction falls inside accepted risk; redact when a specific data element pushes it over; block when it exceeds tolerance; rate limit when volume or repetition raises likelihood; and escalate for human approval when the score sits in a band that requires a named decision.

The unit of enforcement matters. A system rated acceptable for drafting text may carry a different impact when it can call a tool that writes to a record or sends data externally. Evaluating policy per interaction and per tool call lets the autonomy dimension of your impact scale produce different outcomes for different actions.

Each decision should leave evidence: the policy and criteria version applied, the inputs evaluated, the action taken, and the approver for any escalation. That record lets you answer an auditor's question about which criteria governed a specific decision.

Where Trussed fits

Trussed enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants. Policy is enforced at runtime, not reviewed after the fact, with the actions allow, redact, block, rate limit and escalate for human approval, applied to individual interactions and tool calls. That enforcement produces continuous audit-ready evidence, mapped to frameworks including NIST and the EU AI Act. Deployment is self-managed in the customer cloud, with configurable retention and data residency.

Trussed does not set your risk criteria. That remains a decision for your risk function. It gives a regulated enterprise a place to apply those thresholds and a record of what was allowed and what happened.