Questions to Ask SaaS AI Vendors About Prompt Data: A CIO Guide for Health Systems
Ask every SaaS AI vendor to trace a prompt from entry to deletion: which parties receive it, whether each is covered by a BAA, how long prompts, outputs, and logs persist, whether any of it is used for training or human review, and what evidence proves it. Treat contract language as a claim, and accept only answers the vendor can demonstrate.
Why the prompt path is usually the unclear part
Prompts enter AI systems from several directions: a clinician pasting a note into a chat window, a staff member summarizing a message, or a SaaS feature that quietly adds record context to a request. Any of these can carry PHI. Security questionnaires and BAAs tend to describe the vendor's controls in general terms, while the actual route of a prompt through application code, model providers, and storage is left implicit.
Two regulatory points frame the review. HHS states that a BAA cannot authorize a business associate to use or further disclose PHI in a way that would violate the Privacy Rule if the covered entity did it, with narrow exceptions for the business associate's proper management and administration and for data aggregation services. And HHS preamble text in the 2025 Security Rule rulemaking states that ePHI in AI training data, prediction models, and algorithm data maintained by a regulated entity for covered functions is protected by the HIPAA Rules. The same text notes that generative AI tools have produced outputs containing names and personal information of people in their training data, and that training on patient data could result in impermissible uses and disclosures.
The practical consequence is that the review should follow the data, not the document. The sections below are organized by the lifecycle of a prompt, and each question is paired with what a strong answer looks like.
Data flow, subprocessors, and BAA scope
Start with the path. Ask: which entities receive, store, or process prompts and outputs that may contain PHI? That list should include the vendor, every subprocessor, and every model provider behind the product. A strong answer is a current data-flow diagram that names each party, states what data each one receives, and identifies which are covered by a BAA. HHS states that covered entities and business associates must obtain satisfactory assurances, in the form of a BAA, that a cloud service provider will appropriately safeguard the PHI it creates, receives, maintains, or transmits. A model provider that sees PHI is in that position.
Ask next about the entry point. What does the product capture before sending a request: only the user's text, or also attachments, record context, and system instructions? Embedded features deserve specific attention because users may not know what the feature sends on their behalf. Ask the vendor to demonstrate a request from the UI and show the payload that leaves the application.
Then test BAA scope. Does the BAA cover every service and feature that can receive PHI, or only the core product? Does any vendor right to use data for its own management and administration or for aggregation extend to prompts? Compare that wording with the Privacy Rule limits above. Finally, ask how the vendor notifies customers when a subprocessor or model provider is added or replaced, and whether the customer can object before PHI flows to it.
A useful risk-analysis habit, drawn from the HHS preamble questions, is to confirm you have identified all ePHI your organization creates, receives, maintains, or transmits, and whether each vendor or consultant does the same on your behalf.
Retention, deletion, training, and human review
Storage is where prompt data outlives the interaction. Ask for retention periods for each artifact separately: prompts, outputs, application logs, traces, abuse or quality monitoring stores, and backups. A strong answer gives a number for each, identifies who can change it, and explains how deletion propagates to subprocessors and backups. Ask the vendor to demonstrate a deletion and to show what record exists that it happened.
Training use is a distinct disclosure path and needs a direct question: is customer prompt or output data used to train, fine-tune, or evaluate any model, by the vendor or by a model provider? The FTC has stated that model-as-a-service companies that fail to abide by their privacy commitments may be liable, including commitments not to use customer data for secret purposes such as training or updating models, directly or through workarounds. It also notes these companies have a business incentive to ingest more data, which can conflict with obligations to protect it. Ask for the commitment in writing, ask what technical control enforces it, and ask what happens to your data if the vendor changes its policy.
Human review is a separate question from training. Ask whether any person, at the vendor or a provider, can read prompts for abuse monitoring, quality review, or support, under what conditions, and whether those staff are bound by the BAA. A strong answer describes the conditions and the access record, not only the policy.
For your own benchmark, CMS internal guidance recommends versioning production prompts and tracking traces and evaluations, and keeping records of input prompts and model configurations. That guidance is written for CMS users and is not a health system requirement, but it is a reasonable standard for asking whether you could reconstruct what was sent.
Residency, access controls, incident notification, and audit evidence
Ask in which regions prompts are processed and stored, including by subprocessors and for backups, and whether the customer can restrict regions contractually and technically. A strong answer distinguishes where data is stored from where it is processed, and names any exceptions such as support access from other locations.
For access, ask who can view prompts and outputs, how access is granted and reviewed, and whether customer-side administrators can see the same access record. Ask how the vendor will notify you of an incident involving prompt data, how quickly, and whether the notice extends to incidents at subprocessors and model providers.
Audit evidence ties the rest together. The Security Rule's technical safeguards at 45 CFR 164.312 apply to both covered entities and business associates, which makes audit-control evidence a reasonable request of the vendor and something your own environment should also produce. Ask what records exist of prompts sent, to which downstream party, and when; whether the BAA allows the health system to audit the vendor's security practices; and what the vendor will provide on request, in what format, and over what period. Vendor-held records are useful, but they describe the vendor's view. Records generated on your side of the boundary describe what actually left.
Telling a verifiable answer from a contractual assurance
A contractual assurance states what the vendor promises. A verifiable answer is one you can observe, test, or reproduce. Both have value, but only the second reduces uncertainty before signature. Use this contrast to score responses.
- Subprocessors: Assurance: a statement that all subprocessors are under agreements. Verifiable: a named list, a data-flow diagram, and a BAA status for each party that sees PHI.
- Training use: Assurance: a clause that customer data is not used for training. Verifiable: the clause plus a described technical control and a defined notice process for policy changes.
- Retention: Assurance: data is deleted on request. Verifiable: stated periods per artifact, a live deletion demonstration, and a record that deletion occurred.
- Human review: Assurance: access is restricted. Verifiable: written conditions for review and an access record that you can request.
- Residency: Assurance: data stays in a named region. Verifiable: stated storage and processing locations for every party in the path, including backups and support access.
- Audit evidence: Assurance: logs are available on request. Verifiable: a sample record showing what was sent, to whom, and when, in a format you can retain.
Where customer-side enforcement fits
Vendor answers describe what happens after a prompt arrives. They cannot limit what you send. A control inside your own boundary can, which is why it is worth asking what you can enforce before a prompt leaves your environment: allow it, redact the sensitive elements, or block it. This follows the data minimization principle in CMS technical guidance, which recommends that AI tools access only strictly necessary information, and it applies the same default of keeping data within authorization boundaries to external AI vendors.
Trussed enforces policy in the execution path, so PHI can be redacted or blocked before a prompt reaches a vendor, and enforcement is applied at runtime, not reviewed after the fact. Each enforcement decision produces audit-ready evidence mapped to HIPAA. It can be deployed self-managed in the customer cloud, with configurable retention and data residency. Trussed runs in production with healthcare and pharma organizations.
The tradeoff is scope. Runtime enforcement governs interactions that pass through the path, so it must cover the tools, features, and assistants your staff actually use. It does not replace the vendor review above: whatever you allow through still lands with a vendor whose retention, training, and subprocessor practices need to be verified.
The lifecycle of a prompt
Entry
What a clinician, staff member, or embedded feature captures and sends.
Path
Which vendor systems, subprocessors, and model providers receive it.
Storage
What is retained, where, for how long, and whether it feeds training.
Access
Who can see prompts and outputs, including support and review staff.
Evidence
What the health system can obtain afterward to prove what happened.
Review how prompt data leaves your environment
Walk through your highest-risk AI workflows with a Trussed expert and see how in-path enforcement and audit-ready evidence apply to your vendor review.
Request a DemoHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.