Run the exercise in six steps
The mock examination follows one sequence, from scoping through remediation. The table summarizes each stage; the sections that follow explain how to carry them out.
| Step | Stage | What happens |
|---|---|---|
| 1 | Scope | AI systems, generative AI, and agents mapped to the frameworks you will test against. |
| 2 | Request | Examiner-style request list issued with deadlines. |
| 3 | Test | Evidence checked for completeness, accuracy, and timeliness. |
| 4 | Interview | Owners walk through live use cases. |
| 5 | Rate | Findings rated on a severity scale defined in advance. |
| 6 | Remediate | Owners, dates, and retesting tracked to closure. |
Define scope and the frameworks you will test against
Start by defining scope and the applicable supervisory expectations. Scope covers AI systems, generative AI, and AI agents, each mapped to the frameworks you will test against.
Agree on this mapping before any requests go out. It sets the boundaries of the exercise and gives every later finding a clear reference point.
Assemble the team and assign roles
Because the bank plays the examiner, the exercise needs people in examiner-style roles who issue requests, test evidence, and rate findings. Owners of the AI systems and use cases are the counterparty: they produce evidence and are interviewed.
- Examiner team: issues the request list, tests the evidence, and rates the gaps.
- Owners: produce the requested evidence and walk through live use cases.
- Remediation owners: take named responsibility, with dates, for each finding.
Build the request list and test the evidence
Issue an examiner-style request list with deadlines. The deadlines matter because the exercise measures whether evidence can be produced when asked.
Check each item three ways
- Completeness: the evidence covers what was requested.
- Accuracy: the evidence reflects how the system actually operates.
- Timeliness: the evidence arrives within the deadline set.
Interview the owners
After testing the evidence, have owners walk through live use cases. The interviews show whether what the owners describe matches what the evidence shows.
Probe runtime and audit evidence for AI agents
For AI agents, the request list should reach beyond documentation into runtime and audit records. The exercise should confirm that the bank can produce each of the following:
- Identity: records showing which agent acted.
- Permissions: records showing what each agent was allowed to do.
- Tool-call logs: logs of the tools each agent invoked.
- Policy enforcement records: records showing that policies were enforced at runtime.
If any of these cannot be produced, accurately and on time, treat it as a gap and carry it into the rating step.
Rate findings and track remediation
Rate findings on a severity scale defined in advance, so that ratings are consistent and are not negotiated after the fact.
Then track remediation to closure. For each finding, record the owner, the target date, and the retest that confirms the gap is closed.