Enforcing per-tool PHI policy on LangChain FHIR and EHR tools
Treat every LangChain FHIR or EHR tool as its own policy boundary. Evaluate each call before it reaches the API and evaluate the returned output before the agent or another tool sees it. Key the policy on tool identity, FHIR resource type, operation, fields, caller role and purpose. Decide allow, redact, block, rate limit or escalate for approval, and write an evidence record for every decision.
Where enforcement sits in the tool-call path
- 1
1. Resolve identity
Establish the authenticated user, role and stated purpose behind the call. A single shared credential should not be the only control.
- 2
2. Evaluate the request
Check tool identity, resource type, operation and requested fields against that tool's policy before anything reaches the FHIR or EHR API.
- 3
3. Apply the pre-call decision
Allow, block, rate limit or hold for human approval. Blocked calls never reach the API.
- 4
4. Evaluate the response
Inspect the returned JSON, including nested elements and free-text fields, and redact or block what policy does not permit for this role and purpose.
- 5
5. Check onward flow
Treat data passed from one tool's output into another tool's input, or into the final answer, as a new evaluation, not an inherited permission.
- 6
6. Write the evidence record
Record the decision and its inputs at the moment of enforcement.
What each decision should record
- Tool name and version, with a timestamp
- Authenticated caller, role and stated purpose
- FHIR resource types and operation requested
- Decision outcome: allow, redact, block, rate limit or escalate, with the policy rule that fired
- Fields redacted or withheld from the response
- Approver identity and outcome where a human approved or denied the call
Why app-wide permissions and prompts are not enough
Many LangChain agents that reach FHIR or EHR systems run with one service credential and a system prompt asking the model to be careful with patient data. Neither control operates on the individual call. A credential that can search patients, read clinical notes and write back orders grants all of that to every task, whatever the user or purpose. A prompt instruction is a request to the model, not an enforced rule, and it can be overridden by retrieved content that the model treats as instructions.
The tools differ in sensitivity. Patient search returns identifiers and demographics. Clinical notes retrieval can return free text with much more. Medication history reveals conditions by inference. Scheduling touches appointment details. Write-back changes the record itself. FHIR models data as discrete resources such as Patient, Observation and MedicationRequest, accessed through create, read, update and delete operations, so a policy can distinguish a read of one resource type from an update of another.
Three failure modes follow from the missing boundary: the agent retrieves more than the task needs, PHI from one tool's output flows into another tool's input, and the final response includes data the caller's role should not see. Each is a design risk to test for in your own environment. The remedy is the same: a decision made on every call, in the execution path.
Building the per-tool policy model
Start with an inventory of every FHIR and EHR tool the agent can reach: which resource types each can return or modify, which operations it performs and which fields appear in its output. Tools that wrap broad endpoints often return more than their descriptions suggest, so inspect real payloads.
Then express policy against six dimensions. Tool identity says which boundary applies. Resource type and operation separate a Patient read from a MedicationRequest update. Data fields say which elements may pass, which are redacted and which cause a block. Caller role and purpose tie the call to the minimum-necessary idea: each call should carry only the data that role needs for that task. Field lists and thresholds are organizational choices, so have privacy and compliance owners set them rather than copying defaults.
Test redaction on real FHIR JSON. Redaction that handles top-level fields but misses nested extensions or narrative text gives false assurance. Free-text elements, such as note content, need explicit treatment because identifiers can appear anywhere in them.
- Tool identity: Each tool gets its own policy, not a shared application-level rule set.
- Resource type and operation: For example Patient, Observation, MedicationRequest or DocumentReference, crossed with read or write.
- Fields: Allowed, redacted or blocked elements, including nested and free-text content.
- Caller role and purpose: The same tool may be allowed for one role and task and blocked for another.
Example starting points by tool class
These are illustrative design defaults, not standard values. Adjust them to your data, roles and risk assessment.
- Patient search: Allow for scheduling and intake roles. Rate limit to curb bulk enumeration. Redact fields beyond what matching requires.
- Clinical notes retrieval: Allow for treating roles with a stated purpose. Redact identifiers in free text. Block for roles with no clinical need.
- Medication history: Allow for clinical and pharmacy roles. Restrict to the requested patient and the needed elements. Block onward passing to unrelated tools.
- Scheduling: Allow with limited fields. Redact clinical detail from appointment data.
- Write-back: Highest risk class. Escalate for human approval, restrict to specific resource types and block deletes by default.
- Any tool triggered by untrusted content: Apply the same policy regardless of how the model arrived at the call. Prompt wording never widens permissions.
Tradeoffs and where runtime enforcement fits
Per-call enforcement adds latency and a policy set to maintain. Redaction can reduce the usefulness of an answer, and a task denied for lack of purpose may need a clearer request path. Approval queues slow write-back, which is the intended effect for the highest-risk tool class, but they need an owner and a response expectation. Keep policies small, version them and review them when a tool's output schema changes. Also note that ONC (g)(10) conformance testing covers standardized FHIR API behavior, not whether an agent honors minimum-necessary limits, so API certification does not replace this control.
Trussed enforces policy in the execution path across agents, tools and MCP servers. It applies allow, redact, block, rate limit and escalate-for-approval decisions at runtime, not reviewed after the fact, and generates audit-ready evidence from that enforcement. Its scope is individual interactions and tool calls. Trussed runs in production with healthcare organizations. The policy model described above is useful whichever mechanism you use to enforce it.
Policy decision on each tool call
Before the call
Check tool, caller role, purpose, resource type and operation against policy.
After the response
Redact or block fields in the returned FHIR payload before the agent sees them.
Decision
Allow, redact, block, rate limit or escalate for human approval.
Evidence
One record per call showing what was permitted and what happened.
See per-tool enforcement and evidence in practice
Walk through how policy decisions are enforced on individual tool calls and how audit-ready evidence is produced for each one.
See How Evidence Is ProducedHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.