Why standard reference calls miss runtime governance
Open-ended reference calls tend to collect opinions about what a product can do. A structured reference check works differently: every reference customer is asked about the same control domains, redacted artifacts are requested instead of opinions, and the answers are scored against your own control and regulatory requirements.
The table below lists the six control domains used throughout this guide, what to ask about in each, and the artifact to request.
| Control domain | What to ask about | Evidence to request |
|---|---|---|
| Deployment scope | Production versus pilot use, duration, and breadth. | Production dates, volumes, and named processes. |
| Runtime enforcement | Blocked actions, overrides, false positives, latency. | Redacted enforcement reports. |
| Agent access | Identity, least privilege, and permission review. | Redacted permission inventories. |
| Tool calls and MCP | Approval, monitoring, and restriction of connections. | Redacted approval records. |
| Audit evidence | Log completeness, export, and retention. | Redacted log exports. |
| Change, incidents, exams | Operational handling and regulatory support. | Named processes and records tied to your regulatory requirements. |
Deployment scope and runtime policy enforcement
Deployment scope
Ask whether the reference uses the product in production or in a pilot, how long it has been in use, and how broad the deployment is. Production dates and volumes separate a deployment-backed answer from a general statement.
Runtime enforcement
Ask about the following, and request redacted enforcement reports to support the answers:
- Blocked actions
- Overrides
- False positives
- Latency
Agent identity, least privilege, tool calls, and MCP
Agent access
Ask how agents are given an identity, how least privilege is applied, and how permissions are reviewed. Request redacted permission inventories rather than a description of intent.
Tool calls and MCP
Ask how connections are approved, monitored, and restricted. Redacted approval records show whether the process is in use or only described.
Audit evidence, change management, incidents, and examination support
Audit evidence
Ask about log completeness, export, and retention, and request redacted log exports so the answer can be checked against what the reference actually holds.
Change management, incidents, and examinations
Ask how the reference handles change and incidents operationally, and how the vendor has supported regulatory examinations. Tie each question to one of your own regulatory requirements.
Deployment-backed answers compared with marketing claims
Record the basis of each answer so that references can be compared on the same terms.
| Answer basis | What it looks like |
|---|---|
| Production evidence | Production dates, volumes, and logs. Deployment-backed. |
| Described process | A named process the reference describes. |
| General claim | A general statement about what a product can do. Not deployment-backed. |
Running the reference check
- Define your control and regulatory requirements before the call and tie each question to one.
- Ask the vendor for references that match your scale, industry, and use of agents.
- Use the same question set for every reference so answers can be compared.
- Request redacted artifacts: enforcement reports, permission inventories, approval records, and log exports.
- Record whether each answer is backed by production evidence, a described process, or a general claim.
- Escalate unresolved gaps to the vendor in writing and track responses before selection.