Build vs. Buy for LLM Governance in a Health System

Why a binary build-or-buy framing fails

Health systems are adopting LLMs across clinical, operational and administrative workflows, and each workflow can touch PHI. The HIPAA Security Rule requires regulated entities to maintain reasonable and appropriate administrative, physical and technical safeguards for the confidentiality, integrity and availability of ePHI. It also requires protection against reasonably anticipated threats and against unauthorized uses or disclosures. An LLM workflow that creates, receives, maintains or transmits PHI should be scoped as an ePHI-handling system, whatever the model behind it.

A single build-or-buy verdict hides the fact that governance contains parts with very different characteristics. Some parts are organizational decisions. Others are engineering systems that must run continuously in the path of every request. Splitting the problem into four components lets each be judged on its own terms: policy definition, runtime enforcement, evidence generation and retention, and deployment and data residency.

What in-house runtime enforcement actually requires

Enforcement only counts if it sits in the execution path. A review that happens after the fact can describe a problem, but it cannot prevent PHI from reaching a model or stop a tool call from running. An in-house layer therefore has to intercept each interaction and each tool call, evaluate it against policy, and apply one of a small set of actions: allow, redact, block, rate limit or escalate for human approval.

The hard part is consistency. A health system rarely runs one model. It runs several models, agents built by different teams, a growing set of tools and, increasingly, MCP servers. If each integration implements its own checks, policy drifts. A PHI rule that is redacted in one application can be missed in another. Based on architectural experience rather than sourced benchmarks, internal builds tend to stall at this point. The first application is enforced well, and each additional model, agent or tool adds adapter work, policy exceptions and testing.

Agent authorization adds a further requirement. NIST lists short-lived, dynamic and revocable authorization among its design tenets for agent identity management. An in-house layer should be able to issue narrow permissions for agent and tool access and withdraw them quickly. Human-approval escalation also needs real workflow behind it: routing, timeouts, a record of who approved what, and a defined outcome when no one responds.

Evidence is a continuing operation, not a deliverable

Teams often budget for enforcement and treat evidence as a reporting task to finish later. In practice, evidence has to be generated by enforcement itself, at the moment a decision is made, and then kept in a form an auditor can examine. That means recording the policy that applied, the action taken and the outcome for each interaction and tool call, protecting those records from alteration, and retaining them under rules the health system sets.

The mapping work does not end at launch. The audit controls provision at 45 CFR 164.312(b), the HIPAA technical safeguards, NIST SP 800-66 Rev. 2 as a Security Rule implementation resource, and the NIST AI RMF all need to be reflected in the controls and the evidence. NIST describes the AI RMF govern function as continual over an AI system's lifespan and across the organization. HHS also published a proposed rule in January 2025 to strengthen the Security Rule, so documentation should record which version of the rule a control mapping was built against.

Whoever owns this, internal team or vendor, owns a standing function. Storage durability, retention configuration, residency, export for auditors and remapping as requirements change are all recurring work. Estimating that burden for an in-house build is a matter of internal judgment, and it should be done explicitly before committing.

Component-level build-or-buy tradeoffs

Apply the same questions to each component: how specific is it to the organization, how continuously must it run, and how many surfaces must it cover?

• Policy definition: Leans build. Rules about PHI, permitted tools and approval thresholds reflect the health system's own risk decisions and need privacy and security officer ownership. A purchased layer should accept these policies, not replace the decision.
• Runtime enforcement: Leans buy as model, agent, tool and MCP server counts grow. Building is viable for a narrow, single-model use case. Cost of ownership rises with every added surface and every policy change that must stay consistent.
• Evidence generation and retention: Leans buy unless the team can sustain durable storage, retention controls and ongoing HIPAA and NIST mapping. Evidence produced separately from enforcement tends to fragment.
• Deployment and data residency: A constraint, not a preference. If PHI-adjacent data cannot leave the health system's environment, only self-managed options in its own cloud qualify, whether built or bought.

Where a purchased runtime control layer fits

A purchased layer is warranted when the number of models, agents and tools is growing, when more than one team builds AI workflows, and when the organization needs consistent enforcement and continuous evidence without staffing a dedicated platform team. It is less compelling for a single, narrow workflow with stable requirements, where an in-house build may be proportionate.

Trussed is one example of the category. It enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants, with actions to allow, redact, block, rate limit or escalate for human approval, scoped to individual interactions and tool calls. Enforcement produces continuous audit-ready evidence mapped to HIPAA and NIST, among other frameworks. It deploys self-managed in the customer cloud with configurable retention and data residency. It runs in production with healthcare organizations, and has run continuously for 18+ months in a regulated enterprise at 99.95% uptime on live production traffic, with 100+ models under one policy plane.

Whichever path you choose, keep policy ownership internal, assign a named owner for evidence and mapping upkeep, and re-run this component-level assessment as the model and tool estate changes.