Section 1557 Disparity Testing on Google ADK Agents Before Go-Live
Section 1557 disparity testing for AI agents means running controlled, counterfactual test cases against an agent in pre-production, measuring outcome and tool-call differences across protected characteristics, comparing them with documented thresholds, and keeping the plan, results and remediation as evidence of reasonable mitigation efforts under 45 CFR 92.210(c). It is a pre-deployment practice. It does not cover what happens to live interactions after launch.
Step 3: Run the plan in pre-production
- 1
Freeze the build
Record agent version, model, instructions, tool definitions and sub-agent configuration.
- 2
Execute cohort pairs
Run each counterfactual set repeatedly and capture outputs, tool calls and state.
- 3
Compute metrics
Calculate parity ratios or effect sizes for outcomes, and compare tool selection and arguments across cohorts.
- 4
Apply thresholds
Mark each result pass or fail against the thresholds set before the run.
- 5
Remediate and retest
Adjust instructions, tools or routing, then rerun the full plan on the new build.
- 6
Approve go-live
A named owner signs off on the results or documents accepted residual risk.
What Section 1557 expects, and what it does not specify
Section 1557 prohibits discrimination on the basis of race, color, national origin, sex, age or disability in certain health programs and activities. Under 45 CFR 92.210(c), for each patient care decision support tool a covered entity identifies under paragraph (b), it must make reasonable efforts to mitigate the risk of discrimination resulting from the tool's use in its health programs or activities. As of November 2026, that text appears in the current eCFR. The compliance table in 45 CFR 92.1 requires compliance with 92.210(b) and (c) within 300 days of July 5, 2024.
The standard is reasonable efforts, not zero disparity. The rule does not prescribe a test method, a metric or a numeric threshold. That leaves the engineering team to choose a defensible method and to be able to show why it was chosen. Whether a particular component or vendor is itself covered depends on its funding and contractual relationship, so confirm that with counsel rather than assuming it.
Step 1: Scope the ADK agent against 92.210(b)
Start from the regulatory text of 92.210(b) as published on eCFR, then inventory what your agent actually does. An ADK agent is more than a model call. It combines instructions, tools, sub-agents and session state, and each can shape a patient-facing or clinician-facing outcome. Record each behavior that informs or supports a clinical or patient care decision, then decide with compliance whether it is an identified tool. Behaviors that are clearly administrative can be documented as out of scope, with the reasoning.
- Root agent responses: Final answers, recommendations or rankings a clinician or patient may act on.
- Tool calls: Which tool is selected, with what arguments, for which input, such as record lookups, risk scoring or scheduling.
- Sub-agent delegation: Routing decisions that send some users to different specialists or workflows.
- Session state: Attributes carried between turns that could act as protected characteristics or proxies.
- Resource allocation logic: Any step that prioritizes, screens or limits access to services.
Step 2: Design cohorts and choose metrics
Use counterfactual testing. Hold the task constant and vary only the demographic group, language or dialect in otherwise identical inputs. Cover the protected bases in the rule (race, color, national origin, sex, age, disability) and the proxies that appear in your data, such as language, dialect or phrasing that signals disability. Build each pair from realistic scenarios drawn from the agent's intended use, and keep clinical facts identical across variants.
For multi-step agents, apply the same discipline to session state. Seed identical conversation history and change only one attribute, then check whether a sub-agent or tool downstream behaves differently. Because LLM output varies between runs, repeat each case enough times to separate real differences from noise.
Two documented metric options are available. The first is a positive-outcome parity check, where the probability of a positive outcome should be similar across groups, with a common reference threshold of 0.8 (the four-fifths rule). The second is effect size, where bands of 0.2, 0.5 and 0.8 are conventionally read as small, medium and large differences. These are general conventions, not Section 1557 requirements. Pick one or both, set the pass/fail threshold before running the tests, and write down why it suits the clinical context. Add tool-misuse checks alongside outcome parity: unsafe tool selection, excessive agency and unauthorized action attempts, compared across cohorts.
Step 4: Retain evidence of reasonable efforts
Because the standard is reasonable efforts, the record is what makes the work defensible. No rule text lists required artifacts, so the following is practitioner judgment about what a reviewer would want to see: the scoping decision and its rationale, the cohort definitions and how pairs were constructed, the metrics and thresholds with the justification, the raw results per agent version, the remediation steps taken, the retest results and the go-live approval with the approver's name and date.
Store these in a retained, versioned location so they can be matched to the build that shipped. Any later change to instructions, tools, sub-agents or the underlying model should trigger a rerun of the relevant tests. Test governance can also sit within the broader policies, procedures and training your organization already maintains under the rule.
Where pre-deployment testing stops and runtime control starts
Pre-deployment testing shows how a specific build behaved on the cases you designed. It cannot show how the agent behaves with live inputs, drift over time or a tool configuration added after launch. Those gaps need controls that operate while the agent runs.
Trussed does not perform disparity testing, and it does not replace the work above. Trussed enforces policy in the execution path for individual interactions and tool calls: allow, redact, block, rate limit or escalate for human approval. Test results can inform what those policies say. For example, a tool call that failed a threshold in testing can be routed to human approval at runtime. Because policy is enforced at runtime, not reviewed after the fact, enforcement itself produces audit-ready evidence that a regulated enterprise can retain alongside its pre-go-live test records.
From scope to evidence
Scope
Identify agent behaviors and tool calls that support patient care decisions.
Design
Build counterfactual cohorts, choose metrics and set thresholds.
Run
Execute the plan against the agent in pre-production.
Document
Retain plan, results, remediation and retest records.
Enforce
Apply runtime policy to individual interactions and tool calls after launch.
Turn test findings into enforced policy
See how policy enforcement in the execution path applies to agent interactions and tool calls, and how it produces audit-ready evidence.
See How Evidence Is ProducedHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.