HIPAA Business Associate Risk for LLM Vendors: Assessment Checklist

When an LLM vendor becomes a HIPAA business associate

The starting point is whether the vendor creates, receives, maintains, or transmits PHI for a covered entity or another business associate. In LLM deployments, this review must go beyond the primary model request and include every place PHI may appear or persist.

LLM vendor HIPAA review areas

The review should connect the legal business associate determination to the technical architecture used by the AI system. The following areas organize the core questions without treating the model as the only risk surface.

Map PHI across the LLM architecture before reviewing the contract

Before reviewing a business associate agreement, map where PHI can enter, move, persist, or be transformed inside the LLM architecture. This helps legal, security, procurement, and product teams evaluate the same risk surface.

1. 1

   Prompt path

   Identify whether prompts can contain PHI and whether prompts are stored, logged, cached, or used beyond inference.

2. 2

   Output path

   Determine whether generated responses containing PHI are stored in chat history, tickets, analytics, or downstream systems.

3. 3

   Embedding path

   Assess whether embeddings are derived from PHI and where they are stored, searched, backed up, or deleted.

4. 4

   Agent tool path

   Map every system an AI agent can call, including EHR, CRM, scheduling, claims, or document repositories.

5. 5

   Subprocessor path

   List model providers, hosting providers, vector databases, monitoring systems, and logging tools that may touch PHI.

Business associate agreement checklist for LLM vendors

The assessment should verify the contractual and operational controls that determine whether PHI is handled within approved boundaries.

• Confirm the vendor’s permitted uses of PHI for the approved business purpose.
• Verify downstream BAAs for subprocessors that may create, receive, maintain, or transmit PHI.
• Review retention and deletion requirements for prompts, outputs, embeddings, logs, and caches.
• Confirm Security Rule safeguards that apply to the vendor’s systems and services.
• Apply minimum necessary controls to the PHI fields, records, tools, and systems the AI workflow can access.
• Require auditability before production access is granted.
• Review whether model training or fine-tuning paths may touch PHI.

Technical safeguards that matter for LLM and agent deployments

LLM and agent deployments require controls that operate at the point where prompts, generated responses, embedded data, and connected tools are used. Contract terms are necessary, but they should be supported by enforceable technical safeguards.

• Least privilege: Limit AI agents and applications to the minimum PHI fields, records, and systems needed for the approved use case.
• Runtime policy enforcement: Apply policies at the point of access, including tool approval workflows and restrictions on sensitive actions.
• Audit logging: Record request activity, tool calls, data access events, and policy outcomes for investigation and compliance review.
• Data segregation: Understand whether the vendor uses shared multi-tenant infrastructure, dedicated instances, or other isolation controls.
• Deletion verification: Require evidence that prompts, outputs, embeddings, logs, and caches can be deleted according to contract terms.

Governance process before granting PHI access

A repeatable governance process helps prevent informal pilots or employee experimentation from creating unreviewed PHI exposure. The same review should apply before procurement, pilot access, and production deployment.

• Create an AI vendor intake gate: Require BA status review before procurement, pilot access, or employee use with PHI.
• Document the approved use case: Tie permissions, data fields, retention, and permitted PHI use to a specific business purpose.
• Review changes periodically: Reassess when the vendor changes subprocessors, retention practices, model training terms, or connected tools.
• Separate pilot and production controls: Do not allow pilot exceptions to become production defaults without formal risk approval.
• Preserve evidence: Keep vendor answers, architecture diagrams, BAA terms, logging samples, and approval decisions in an auditable record.

Assessment focus by data path

The same PHI handling question may have different implications depending on whether it appears in prompts, outputs, embeddings, connected tools, logs, or downstream services.

Where runtime governance reduces assessed risk

Runtime governance helps make approved HIPAA controls enforceable in AI systems. It supports least privilege permissions, tool approval workflows, and audit logging so that approved use cases remain bounded after access is granted.

Use runtime governance, least privilege permissions, tool approval workflows, and audit logging to make approved HIPAA controls enforceable in AI systems.