How to Test a KYC Review Agent with Synthetic Identity Documents Under Current Model Risk Guidance
A defensible test plan for a KYC review agent has four parts: a corpus manifest of fully synthetic documents with a documented rationale for each class, scenarios drawn from recognized document-fraud indicators, pass and fail criteria approved before testing begins, and results retained as evidence. Testing shows how the agent behaved on a fixed corpus at one point in time. It does not show what the agent is permitted to do or does once live, so tool calls, personal-data handling and human-approval decisions need runtime policy enforcement and evidence after launch.
What to retain for model risk management review
- Risk-tiering and materiality rationale for the KYC agent
- Corpus manifest with rationale per class and the generation record
- Approved acceptance criteria, with approver and date, fixed before testing
- Raw results per document, with agent version and configuration
- Outlier analysis, comparison to any prior version, and documented limitations
- Open issues, the independent validator's challenge, and the re-test triggers
What the test has to demonstrate
Real customer identity documents are poor test material because of privacy and data-handling constraints. Synthetic documents let a team exercise the agent without exposing customer data, but only if the corpus is built deliberately and the results are kept in a form a model risk management reviewer can challenge.
One terminology point first. In fraud literature, a synthetic identity is an identity a fraudster builds by combining real and fictitious data. A synthetic test document is something else: a fictitious document created by the institution to test its own agent. Keep the two terms separate in the test plan.
As of November 2026, the Federal Reserve has published a document titled Revised Guidance on Model Risk Management (SR2602) in addition to the original SR 11-7 text. The validation plan should cite the version the institution's examiner applies. The revised guidance states that it does not set forth enforceable standards or prescriptive requirements, so frame the evidence as support for sound practice, not as a mandated checklist. The FDIC adopted the FRB and OCC guidance with technical conforming changes, which makes it applicable to certain FDIC-supervised institutions. Applicability depends on the institution's regulator and charter.
The testing language is consistent across the guidance. SR 11-7 says testing covers accuracy, robustness and stability, limitations, and behavior across a range of inputs. It also says testing should include conditions outside ordinary expectations and span the products or applications the model is intended for. It adds that validation before first use should be in line with the potential risk from the model's use. Record the risk-tiering rationale for the KYC agent, because it justifies how much testing you did.
Build the corpus from a manifest, not a folder of samples
Start with a corpus manifest. Each entry names a document class, the reason it is in the corpus, and the expected agent outcome. The rationale is what makes the corpus defensible. Cite the SR 11-7 language on scenarios outside ordinary expectations where it applies, and the document-fraud indicators in NIST SP 800-63A where they apply.
On generation, the method belongs to the institution. Whatever tooling or process creates the documents, use fictitious names, numbers and images, confirm that no real customer data entered the generator or its prompts, and write the method down. A reviewer will ask how you know the corpus is free of personal data.
- Document types: Every identity document type the agent is meant to review, in the layouts and issuers in scope for the product.
- Tampering and forgery patterns: Altered, fabricated and forged variants of each document type, paired with clean originals.
- Quality and edge cases: Damaged, low-resolution, partially occluded and otherwise out-of-distribution documents.
- Escalation triggers: Cases where the correct outcome is referral to a human reviewer, not an automated decision.
- Holdout set: A reserved portion not used while tuning prompts or configuration, consistent with out-of-sample testing.
- Generation record: The method, date and version used to produce each batch, with confirmation that no real customer data was used.
Design scenarios around fraud indicators and escalation
NIST SP 800-63A lists indicators of fraudulent documents: damage, tampering, modification, fabrication and forgery. It also describes facial image comparison against presented documents. These give a reasonable starting taxonomy for scenario categories. Mapping them onto a corpus is a design decision, not a regulatory requirement, so document it as such. NIST SP 800-63A is written for credential service providers, and the institution should treat it as a reference for scenario design, not as a bank KYC rule.
For each category, build graded variants, from obvious to subtle, so the results show where detection degrades. Include clean documents in every batch. An agent that flags everything looks strong on forgery detection and fails on legitimate customers.
Escalation scenarios deserve their own category. Include documents that are ambiguous, internally inconsistent, or outside the supported set, and define the correct outcome in advance. Also state what the corpus cannot test. A synthetic file does not exercise the trust of a capture device, which NIST addresses through sensor authentication or device attestation, so scope that out explicitly and cover it elsewhere.
Set acceptance criteria before the first run
Decide pass and fail criteria before testing and record who approved them. Criteria written after seeing results are hard to defend in front of an examiner. Typical measures are false accept rate on forged and tampered documents, false reject rate on genuine ones, escalation accuracy, and handling of low-quality or out-of-distribution documents. No single threshold applies to every institution. Set values by risk tier and by the cost of each error type. A false accept passes a bad document into onboarding. A false reject adds friction and review workload. Escalation accuracy matters because the safest behavior on an ambiguous document is often to refer it.
If the agent performs face-to-document comparison, check the performance thresholds NIST sets for one-to-one comparison algorithms, using the current NIST text.
When replacing an existing agent or model version, run both on identical inputs and compare against materiality thresholds the institution defines. Include outlier and outcomes analysis in the results package, and record limitations found in testing so ongoing monitoring can track them. The guidance describes ongoing monitoring that regularly assesses model limitations, which is where test findings continue to be used.
Where testing ends and runtime enforcement begins
Pre-production results describe behavior on a fixed corpus at a point in time. They do not show what the agent is permitted to do in production, and they do not record what it actually did on a given customer file. Guidance pairs testing and validation with ongoing monitoring for this reason.
Use test findings to decide which behaviors need runtime policy. Candidates include which tools the agent may call, how personal data is handled in prompts and outputs, and which decisions require human approval. Each can map to an allow, redact, block or escalate decision. For vendor-supplied or updated agents, pin the version and define re-test triggers, so a change is reviewed before it reaches production.
This is the layer Trussed addresses. Trussed is not a testing or red-teaming product. It enforces policy in the execution path, at runtime, not reviewed after the fact, across models, agents, tools and MCP servers. Its scope is individual interactions and tool calls, where it can allow, redact, block, rate limit or escalate for human approval. That enforcement generates audit-ready evidence of what the AI was allowed to do and what it did, mapped to frameworks including NAIC, NYDFS and NIST. Trussed deploys self-managed in the customer cloud with configurable retention and data residency. Insurance is the regulated financial sector where Trussed is deepest, and it supports customers in four regulated verticals: insurance, pharma, healthcare and education.
Pre-production evidence package for a KYC review agent
Corpus manifest
Synthetic document classes, each with a documented rationale.
Scenario coverage
Tampering, forgery, edge cases and escalation triggers.
Acceptance criteria
Pass and fail thresholds fixed and approved before testing.
Runtime handoff
Findings that decide which behaviors are enforced after launch.
Connect test findings to runtime evidence
See how policy enforced in the execution path produces audit-ready evidence for individual interactions and tool calls.
Explore Agent and Tool-Call EnforcementHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.