Key Risk Indicator for AI: Definition, Categories, Thresholds and Evidence

What a KRI means for AI systems

In enterprise risk management, a key risk indicator signals a change in exposure before a loss event occurs. NIST's glossary gives the short form: a metric used to measure risk. NIST AI RMF 1.0 defines risk as the composite of an event's probability and the magnitude of its consequences. An AI indicator is therefore most useful when it can be tied to likelihood, impact, or both.

Many teams already collect AI metrics, but most of them describe performance or adoption. Request volume, latency and accuracy tell you how a system is behaving, not whether exposure is rising. A metric becomes a KRI only when it has a defined risk it speaks to, a threshold, an owner and a required response.

The distinctions below are practitioner framing. They are a working way to keep the terms separate, not a formal standard.

• KRI: Signals change in exposure to a defined risk. Has a threshold, owner and escalation path.
• KPI: Measures progress against a business or performance objective, such as task success or response time.
• KCI: Measures whether a control is operating as designed, such as the share of tool calls evaluated by policy.
• Monitoring metric: Any observed measurement. It becomes a KRI only when tied to a risk, threshold and response.

Categories of indicators relevant to AI use

Choose indicators by risk, not by what is easy to collect. NIST AI RMF 1.0 recommends starting from the risks selected in the MAP function and measuring the most significant first. For AI in production, several categories recur and can be counted at the level of individual interactions and tool calls.

Each category should be tracked as a rate against a baseline, split by application, user group, model and tool. Raw counts rise with traffic and hide real change. Agent security guidance from OWASP names several behaviours worth watching, including drift in approval behaviour, repeated approval bypass attempts, elevated privilege usage, abnormal tool invocation frequency and sudden increases in high-risk actions. That is community guidance, not a regulatory requirement, but it is a practical source of candidate indicators.

• Blocked or redacted interactions: Rate of interactions stopped or modified because they contained restricted data or content.
• Policy violations: Attempted actions that breach a defined policy, grouped by policy and severity.
• Escalations for human approval: Volume, approval rate and time to decision for actions routed to a person.
• Unapproved tool or model usage: Calls to models or tools outside the approved set.
• Approval bypass and privilege signals: Repeated bypass attempts and elevated privilege usage by agents or users.
• Abnormal tool invocation: Unusual call frequency or a sudden rise in high-risk actions against baseline.

Leading, lagging and runtime sources

A lagging indicator reports harm that has already occurred, such as incidents found in a complaint or an audit finding. A leading indicator reports conditions that precede harm, such as a rising rate of attempted policy violations. Risk appetite is usually defended with leading indicators, because they leave time to act.

The source of the data matters as much as the type. Indicators assembled by reviewing activity after the fact lag the interactions and tool calls that create the exposure, and they depend on sampling and on what was retained. Indicators derived from events recorded at the moment of execution cover every evaluated interaction and carry the decision that was actually taken.

This is where runtime policy enforcement becomes a data source. When policy is enforced in the execution path, each decision (allow, redact, block, rate limit, escalate for human approval) is itself an event. Counting those events by category yields indicators without a separate reconstruction step. Trussed enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants, at the level of individual interactions and tool calls, and generates audit-ready evidence from that enforcement.

One caution applies to any source. Enforcement events measure what policy evaluated. An indicator is only as complete as the coverage of the path it sits in, so track coverage as a control indicator alongside the risk indicators.

Setting thresholds, ownership and escalation

NIST AI RMF 1.0 states that it does not prescribe risk tolerance. The organisation must set its own tolerance and appetite, and each threshold should trace back to them. A threshold with no link to a stated appetite is difficult to defend in an audit.

Baselines come first. AI RMF 1.0 calls for baseline metrics for comparison, including a human baseline where AI augments or replaces human activity. Change-based indicators, such as drift in approval behaviour, cannot be detected without a measured starting point. Run in observation mode long enough to understand normal variation before fixing limits.

Decide whether each indicator tracks inherent risk, residual risk, or both. Residual risk is what remains after treatment, so an indicator measured after enforcement describes a different position from one measured before it. State which one is reported.

Use tiered thresholds so that response is proportionate. A warning level prompts review by the indicator owner. A breach level triggers escalation to a named risk owner and a defined action, such as tightening a policy, restricting a tool or pausing an application.

• Define: State the risk, the indicator, its formula and its data source.
• Baseline: Measure normal values, including a human baseline where relevant.
• Set tiers: Set warning and breach levels linked to risk appetite and approved by the risk owner.
• Assign: Name an owner for monitoring and a separate authority for escalation decisions.
• Respond: Document the required action and time to respond at each tier.
• Review: Revisit thresholds on a schedule and after policy or model changes.

Evidence an auditor can inspect

An auditor will ask three things: how the indicator is calculated, whether the underlying data is complete and unaltered, and what happened when a threshold was crossed. Each needs a record.

OWASP's agent security guidance recommends structured decision metadata for high-risk actions: action classification, risk score where applicable, authorisation outcome, approval identifier, execution result and policy version. These fields work as a starting schema for audit-usable events. Recording the policy version with each decision lets you tell whether an indicator moved because behaviour changed or because policy changed. The approval identifier and authorisation outcome allow human escalation indicators, such as volume and bypass attempts, to be reconstructed from event data.

Beyond event records, keep the indicator definitions, threshold approvals, threshold change history, breach notifications and the actions taken in response. Retention and data residency should be configurable to match your obligations. Trussed's enforcement generates continuous audit-ready evidence mapped to HIPAA, NAIC, NYDFS, the EU AI Act and NIST, and can be deployed self-managed in the customer cloud. Whatever tooling you use, test the evidence by selecting one breach and tracing it from the indicator back to the individual events and the response.