LLM Governance Proof of Concept Success Criteria for Bank Model Risk Review

Treat the POC as a miniature model risk review

Many governance proofs of concept are scoped around demonstrations. A demonstration shows that a control can work. A model risk review asks whether the control worked, on which interactions, under whose policy, and what record proves it. If those questions are not settled before the pilot, the result is predictable: stalled approvals and a second pilot to collect what the first one missed.

The current reference point is the Revised Guidance on Model Risk Management issued by the Federal Reserve, OCC and FDIC (SR 26-2, dated April 17, 2026, as of October 2026). Many bank policies still cite SR 11-7, the 2011 guidance. Treat that document as historical context and check that your model risk policy cites the current guidance.

Three ideas in the revised guidance shape POC design. First, effective challenge is critical analysis by objective experts who evaluate model risk and effect appropriate changes across the model lifecycle. The POC should therefore leave artifacts a validator can analyze independently of the POC team. Second, effective policies define expectations and a framework for applying practices commensurate with risk. Success criteria should be written and approved before the pilot, and rigor should scale with the use case's risk profile and usage. Third, a bank's own risk management and governance practices should guide controls for tools, processes or systems the guidance does not cover. Where the guidance is silent, your internal standards set the bar, and the POC criteria should cite them.

Score outcomes the way a validator will challenge them

NIST AI RMF MEASURE 2.3 calls for performance or assurance criteria to be measured and demonstrated under conditions similar to the deployment setting. For a POC, that means the test set should reflect the prompts, tools and data the use case will actually see, not a curated demo script. It also means the inventory in criterion 1 should match what is exercised in criterion 6. A tool the POC never calls is a tool the POC says nothing about.

Define scoring rules before testing. OWASP guidance advises scoring with model-generated refusals taken into account, not classifying answers by matching refusal phrases, and not counting empty responses as successful completions. Apply the same discipline to block and allow outcomes. Decide in advance whether a blocked request that returns an empty reply is a pass, and whether a redaction that removes only part of a sensitive value counts.

For prompt injection, set the criterion as mitigation, not elimination. OWASP states there is no fool-proof prevention within the LLM, only measures that reduce impact. A defensible pass condition is that policy is applied to the interaction or tool call that an injected instruction would trigger, and that the record allows retrospective review of cases that got through. Residual risk should be stated in the POC report, not omitted from it.

What the POC can demonstrate and what it cannot

Keeping this boundary explicit protects the POC from being read as more than it is. Runtime enforcement evidence addresses how policy was applied in operation. Other model risk activities address whether the underlying system is sound.

• Policy applied to individual interactions and tool calls: Demonstrated by the POC through enforcement outcomes and decision records.
• Audit-ready evidence, retention and residency: Demonstrated by the POC through sampled records and retention configuration.
• Human escalation and issue handling: Demonstrated by the POC through approval records and the issue log.
• Conceptual soundness and pre-deployment validation: Not covered. Remains with the validation function. Runtime enforcement evidence does not replace it.
• Model lineage and ownership of the model itself: Not covered. Remains with separate model risk activities and named owners.

Assessing in-path enforcement during the POC

If the POC includes a runtime enforcement layer, the criteria above can be applied to it directly. Trussed enforces policy in the execution path across models, agents, tools and MCP servers, with outcomes of allow, redact, block, rate limit, and escalate for human approval, on individual interactions and tool calls. It generates audit-ready evidence from that enforcement, and it deploys self-managed in the customer cloud with configurable retention and data residency. Policy is enforced at runtime, not reviewed after the fact, so criteria 3 and 4 can be tested against the same decision records.

Trussed does not provide model lineage or dataset provenance, and it does not replace validation or conceptual soundness review. Those stay in your plan as separate line items with owners. Trussed's production experience is deepest in insurance, a regulated financial sector, and the evaluation framework here applies whichever enforcement approach you assess.

Deciding whether the POC is enough to move forward

Use a simple decision rule. If every criterion passes against its pre-agreed threshold, the POC supports moving the use case into the production approval process. If a criterion fails or its artifact is missing, record it as an open issue with an owner rather than adjusting the threshold afterward. Changing criteria after results are known weakens the evidence a validator is asked to rely on.

State the scope limit in the POC report. The revised guidance's governance expectations apply across the model lifecycle, and a POC covers only part of it. The report should list what remains outside the POC, who owns each item, and what must be complete before production approval. It should connect to existing organizational governance and risk controls, and align to data governance for sensitive data, so the pilot does not become a parallel process.

A practical sequence is to draft the six criteria, review them with an independent validator, fix the test set and scoring rules, run the pilot under deployment-like conditions, and assemble the artifacts into a single package for effective challenge.