Implementation Guide | Healthcare

    Testing a LangChain RAG Pipeline for PHI Leakage with Synthetic Patient Data Before Go-Live

    To test a RAG pipeline for PHI leakage, seed a fully synthetic patient corpus into the same retrieval stack you will deploy, then run targeted test cases at each stage: ingestion, indexing, retrieval, prompt construction, generation, and output with citations. Set explicit pass/fail criteria, keep the results as evidence, and treat the exercise as necessary but not sufficient. Pre-go-live tests cannot show behavior on live traffic, so controls that must hold on every request need runtime enforcement.

    Define leakage test cases for each stage

    1. 1

      1. Ingestion

      Verify that patient identifiers and access tags survive chunking and attach to every chunk. Confirm that documents with missing tags are rejected or quarantined rather than indexed as open.

    2. 2

      2. Indexing

      Check collection and namespace scope. Confirm the index does not mix records across tenants or departments in ways the access filter cannot separate.

    3. 3

      3. Retrieval

      Query as users with different entitlements, including a user with no patient relationship. Assert that no returned chunk falls outside the requester's authorization, including near-duplicate and same-name patients.

    4. 4

      4. Prompt construction

      Capture the fully assembled prompt and check it for out-of-scope chunks, metadata fields, and prior-session history.

    5. 5

      5. Generation

      Send direct injection attempts, and seed documents with instructions that tell the model to reveal other records. Probe for re-identification by combining quasi-identifiers in follow-up questions.

    6. 6

      6. Output and citations

      Scan answers, citation text, source titles, and callback or trace payloads for canary values and identifier patterns.

    Where PHI can leak in a RAG pipeline

    A RAG application touches patient content at every stage, and each stage fails differently. Document loaders and text splitters decide what text and metadata enter the index. If access tags or patient identifiers are dropped during chunking, downstream filtering has nothing to enforce. A vector store shared across departments or tenants can return chunks that the requesting user should never see. Prompt templates may concatenate retrieved text, conversation history, and metadata fields into one context. The model can then repeat any of it, and citations can expose source titles or file names that contain identifiers. Callbacks and tracing handlers also receive prompts and retrieved text, so they are an exposure path in their own right.

    The HIPAA Privacy Rule requires appropriate safeguards for PHI and sets limits and conditions on uses and disclosures made without an individual's authorization. That is the baseline against which PHI surfacing in the wrong context is judged. Guidance submitted in a public comment to NIST recommends evaluating retriever and generator components individually and together, and including data ingestion systems, internal APIs, monitoring tools, and model outputs in scope. A sound test plan follows that structure: one test target per stage, plus end-to-end runs.

    Start with a written threat model. List who the requesters are, which records each may see, and which threats apply, such as cross-patient retrieval, prompt injection, and metadata exposure.

    Build and seed the synthetic patient dataset

    Use fully synthetic records rather than de-identified real data. ONC notes that re-identification of anonymized data remains a possible security risk when de-identified clinical data is used to test algorithms and prototypes, and that synthetic health data can provide a no-risk data source for testing needs. Synthea is one option. The ASPE report describes it as generating realistic, but not real, health data for fictitious patients, with modules covering conditions, encounters, laboratory tests, and demographic information, and configuration options for different testing scenarios.

    Structured output alone will not exercise a RAG pipeline well. Add the content your corpus actually contains: clinical notes, discharge summaries, referral letters, and administrative documents such as billing or scheduling text. Embed realistic identifiers in those documents, including names, dates, addresses, phone numbers, and record numbers, in the formats and positions where they appear in real content. Also include edge cases: two patients with the same name, a family member named in another patient's note, an identifier inside a free-text field, and a scanned-document style layout.

    Plant unique canary values, such as an identifier that exists in exactly one synthetic record, so any appearance in the wrong output is unambiguous. Validate the dataset before use, checking individual records and overall statistical content. The Synthea module developers did both, so the corpus is realistic enough to produce meaningful results. Then ingest it through the same loaders, splitters, embedding model, and vector store you will deploy, with the same access metadata scheme.

    Set pass/fail criteria and record the evidence

    Retrieval and prompt assembly are deterministic enough for strict assertions. The pass criterion is zero out-of-scope chunks for every test identity, and zero canary values from unauthorized records in any assembled prompt. Generation is not deterministic, so repeat each adversarial case multiple times and with paraphrased variants. Treat any appearance of an out-of-scope canary or identifier in an answer, citation, or trace as a failure, not a score to be averaged. Decide in advance how you will handle partial exposures, such as a date or a first name without other context, so that the rule is not negotiated after a failure.

    Record enough to reproduce each run. That means the dataset version and generation configuration, the pipeline configuration and component versions, the threat model, each test case with its identity and input, the raw outputs, the pass/fail result, defects found, and the retest after each fix. State plainly that the corpus contained only synthetic records and how it was validated. This gives an auditor and a later reviewer a defensible account of what was tested and what the results were.

    As of November 2026, the HIPAA Security Rule technical safeguards at 45 CFR 164.312 apply to covered entities and business associates, in accordance with 164.306. Confirm the exact audit control wording of 164.312(b) in the current text and map your evidence to it with your compliance team.

    What pre-go-live testing cannot show, and where runtime enforcement fits

    A test run is a sample. It covers the queries, identities, and documents you thought of, against a corpus that will change after launch. It does not show how the pipeline behaves with live traffic, new documents, new users, or novel injection phrasing. For that reason, decide before go-live which controls must hold on every request rather than on the requests you tested. Typical candidates are per-user access decisions on retrieved content, identifier handling in prompts and outputs, and restrictions on what tools and retrievers may return.

    This is where policy enforcement in the execution path applies. Trussed enforces policy at runtime across models, agents, tools, and MCP servers, applying allow, redact, or block decisions to individual interactions and tool calls, with escalation for human approval where needed. Enforcement is at runtime, not reviewed after the fact. Each decision produces audit-ready evidence mapped to HIPAA, including 164.312(b) audit controls. Trussed runs in production with healthcare organizations, and it can be self-managed in the customer cloud with configurable retention and data residency.

    Trussed is not a testing or red-teaming product. It does not replace the pre-go-live plan above. The two work together: testing finds defects in the pipeline you built, and runtime enforcement limits what a defect or an untested input can do in production.

    PHI leakage test coverage by pipeline stage

    Ingestion and indexing

    Seed synthetic records, canary values, and injection payloads before chunking.

    Retrieval

    Confirm each user identity only retrieves records it is authorized to see.

    Prompt and generation

    Inspect assembled prompts and probe model output for out-of-scope PHI.

    Output and citations

    Scan answers, citations, and metadata for identifiers.

    Runtime

    Enforce allow, redact, or block where testing cannot reach.

    Plan what carries over from testing into production

    Talk through which PHI controls in your RAG pipeline need runtime enforcement and what evidence they should produce.

    Talk to an Expert