How to Build Google ADK Evaluation Sets With Synthetic PHI for HIPAA Pre-Production Testing
Generate synthetic patient records that cover the identifier categories in the HIPAA Safe Harbor provision, verify they trace to no real individual, and convert them into ADK evaluation cases with expected tool trajectories and reference responses. Add PHI-specific cases for identifier leakage, over-broad tool calls and cross-session carryover, and run everything in a non-production environment. Treat the result as a testing control. Live interactions still need runtime policy enforcement.
Step 2: Convert the dataset into ADK evaluation sets
- 1
Define scenarios
List the agent's tasks, such as scheduling, benefits lookup or record summarization, and pick synthetic patients for each.
- 2
Write the inputs
Draft single-turn and multi-turn user inputs that include identifiers in realistic phrasing, including misspellings and partial values.
- 3
Specify expected tool trajectories
Record which tools should be called, in what order, and with which arguments, limited to the identifiers the task needs.
- 4
Write reference responses
State what a correct answer contains and which identifiers it may repeat back to the user.
- 5
Add negative cases
Include requests that should be refused or narrowed, such as asking for another patient's details.
- 6
Version everything
Store evaluation sets, fixtures and agent configuration together so a run can be reproduced.
Why evaluation data should be synthetic
An evaluation set is a repeatable record of what an agent should do for a given input. For a healthcare agent, realistic inputs contain names, dates, contact details and record numbers. If those values come from real patients, the test environment, the evaluation files and every run output become places where real PHI lives, which extends HIPAA exposure to systems that were meant to be disposable.
Synthetic PHI removes that dependency. ONC's HealthIT.gov describes synthetic health data as a no-risk data source that can support testing needs, and describes the Synthea generator as producing records for fictitious patients. That is the source's own characterization, and it is not a guarantee about any dataset you produce. Treat the dataset as an engineering artifact with a documented method and a verification step.
Keep the evaluation data, the agent under test and the run outputs in a non-production environment that is separate from live systems and live data stores. Tools the agent calls during evaluation should point at test fixtures or sandboxes seeded with the same synthetic records.
Step 1: Generate and verify the synthetic dataset
Start with a generator that produces clinically structured patients. Synthea, built from clinical care guidelines and standards of care, is one documented option. Clinical modules alone may not give you identifier-heavy cases, so plan to supplement them with rule-based generation for names, addresses, phone numbers, email addresses and record numbers, plus hand-written adversarial cases.
Map dataset fields to the identifier categories in the Safe Harbor provision of 45 CFR 164.514(b)(2). Read the category list from the regulation text itself, and make sure each category has at least one field that the agent can see or be asked about. Without that coverage you cannot say which identifier types your tests exercised.
Then verify the data. The goal is a dataset in which no record corresponds to a real individual.
- No real seeds: Do not derive values from production records, even after masking. Generate from rules and random draws only.
- Format fidelity: Identifiers should look real enough to trigger the same parsing and matching behavior as production formats.
- Referential consistency: A patient's name, record number and dates must agree across records, tools and conversation turns.
- Collision review: Spot-check generated names and numbers against any real reference data your organization holds, inside a controlled setting.
- Method record: Document the generator, version, parameters and verification results alongside the dataset.
Step 3: Encode PHI failure modes and pass/fail criteria
Generic quality scoring will not tell you whether an agent mishandles identifiers. Add explicit cases for three failure modes and decide the pass criteria before the first run.
Identifier leakage in outputs: the agent repeats an identifier the user did not supply, or repeats one in a context where it is not needed. Pass means the response contains only identifiers the case permits. Fail on any other identifier from the synthetic record, including partial values.
Over-broad tool calls: the agent passes more identifiers than the task requires, or requests a whole record when a single field would do. Pass means tool arguments match the expected trajectory and include no extra identifier fields. Fail on additional arguments, even if the final response looks correct.
Cross-session carryover: after one conversation about patient A, a new session or a new patient B should show no trace of A's identifiers in responses or tool calls. Build paired cases that run in sequence, and treat any appearance of A's values in B's session as a failure. Within a single multi-turn session, also check that the agent keeps identifiers attached to the correct patient.
Set the thresholds for these checks at zero tolerance unless your risk process says otherwise. A leakage check that passes at ninety percent is a documented intermittent failure.
Step 4: Retain artifacts as testing evidence
Evaluation is only useful later if you can show what was tested and how. Retain the synthetic dataset and its generation and verification record, the evaluation set files, the agent version and configuration under test, the pass/fail criteria, the run results, and the dates. Store them with access controls and a retention period set by your own documentation policy.
Your compliance team should decide which HIPAA documentation provisions apply to these artifacts. Do not assume a specific one from this guide. Whatever the answer, a reproducible package lets you rerun the same evaluation after any change to the model, prompts or tools and compare results directly.
What pre-production evaluation does not cover
Evaluation sets measure behavior on cases you anticipated. They cannot cover an identifier format nobody wrote a case for, a prompt injection that arrives through a retrieved document, or a tool call that differs from every trajectory in the set. A passing run is evidence about the tested scenarios, not a control on live traffic.
That gap is the job of runtime enforcement. Trussed enforces policy in the execution path for individual interactions and tool calls across models, agents, tools and MCP servers: allow, redact, block. Policy is enforced at runtime, not reviewed after the fact, and the enforcement produces audit-ready evidence mapped to HIPAA. Trussed runs in production with healthcare organizations. It is not a pre-deployment testing or red-teaming product, so the two controls are complementary: evaluation sets find issues before release, and runtime enforcement applies to what happens once real PHI is in the path.
A practical posture is to feed every failure found in evaluation into policy requirements, and every runtime finding back into new evaluation cases.
From synthetic records to testing evidence
Generate
Synthetic patients with identifier-bearing fields
Verify
Check that no record maps to a real person
Structure
ADK eval cases with expected tool use and responses
Score
Explicit pass/fail criteria for PHI handling
Retain
Datasets, configs and results kept as testing evidence
Pair pre-production testing with runtime enforcement
See how policy is enforced in the execution path for agent and tool calls, and how enforcement produces audit-ready evidence for HIPAA.
Explore Agent and Tool-Call EnforcementHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.