From control telemetry to risk evidence
- BaselineAgent inventory, permissions, tool usage, and incident history before enforcement.
- Control telemetryPolicy decisions, tool-call logs, and agent identity records from runtime enforcement.
- Outcome metricsPermission scope, confirmed violations, near-misses, and incidents.
- ReportingNormalized trends with stated caveats for security and governance reviewers.
Control activity is not risk reduction
Runtime controls produce two different kinds of signal, and they answer different questions. Control activity metrics, such as blocked actions, show whether the controls are working. Risk outcome metrics show whether risk went down: permission scope, confirmed violations, near-misses, and incidents.
Report risk reduction without counting blocked actions. A blocked action is evidence that a control operated, not evidence that risk fell.
Keeping the two metric sets separate prevents a high count of enforcement events from being read as an improvement in risk posture.
Establish the pre-deployment baseline
Capture the baseline before enforcement begins. Without it, post-deployment data has nothing to be compared against. The baseline should cover agent behavior, permissions, and incidents, including:
- Agent inventory
- Permissions granted to each agent
- Tool usage
- Incident history
Choose leading and lagging metrics tied to telemetry
Select leading and lagging metrics from the telemetry your runtime controls actually produce: policy decisions, tool-call logs, and agent identity records. A metric that cannot be traced to a telemetry source cannot be verified later, so tie each one to the records it depends on before enforcement goes live.
Metric categories and the telemetry they require
The table below separates the two metric sets described above and shows the telemetry each depends on.
| Category | Example metrics | What it shows | Telemetry required |
|---|---|---|---|
| Control activity | Blocked actions | Whether the controls are working | Policy decisions and tool-call logs |
| Risk outcome | Permission scope, confirmed violations, near-misses, incidents | Whether risk went down | Permission grants and changes, agent identity records, and tool-call logs, compared with the baseline |
Interpret enforcement data and account for confounders
Before attributing any change to the controls, adjust for the factors that can move the numbers on their own:
- False positives: blocked actions that were not real violations inflate control activity and should not be read as risk avoided.
- Policy changes: a new or revised policy changes what is counted, so note the policy version behind each period.
- Shifts in agent usage: more or fewer agents, or different tool usage, change event volumes regardless of control effectiveness.
- Coverage gaps: agents or actions outside runtime control produce no telemetry, so they need to be identified and tracked separately.
Report results to security and governance stakeholders
Present normalized trends with stated caveats, so reviewers can see both the direction of change and the limits of the data. Show control activity and risk outcomes as separate sets, and compare each against the baseline.
Evidence requirements for governance and audit review
- Each log entry ties an action to a distinct agent identity, not a shared service account.
- Policy decisions record the policy version that produced them.
- Logs are tamper-evident and retained according to a documented retention policy.
- Permission grants and changes are logged with who approved them and when.
- Records can be exported in a form reviewers can query without engineering help.
- Human overrides and approval decisions are captured alongside automated ones.