How to Detect Unapproved AI Use by Clinicians, Broken Down by Department
No single telemetry source shows which unapproved AI models clinicians use. Combine web gateway and DNS data, CASB, endpoint and browser telemetry, and identity logs. Map the activity to departments through directory attributes, classify each destination as approved or unapproved, and rank departments by data sensitivity and clinical context. Detection alone only supports review after the fact. Enforcing policy in the execution path (allow, redact, block, escalate) produces evidence of what was permitted.
- List current telemetry sources and note which departments or device types fall outside each.
- Stand up an approved and unapproved AI inventory, with approved uses stated by department and data type.
- Agree the department attribution method and complete a privacy review before using role or EHR-access attributes.
- Add AI-specific threats to the risk analysis and score departments on PHI likelihood, use type and volume.
- Decide per department which events are allowed, redacted, blocked or escalated, and define the clinician workflow when a request is stopped.
- Confirm with counsel how usage evidence maps to audit controls before relying on it.
Why department-level visibility is the goal
Clinicians in emergency medicine, radiology, oncology, primary care, nursing and revenue cycle may use consumer AI models and AI sites for documentation, summarization, coding or decision support. The risk differs by department because the data and the clinical consequences differ. A coder pasting claim text into a chatbot is a different exposure from a physician pasting a visit narrative with identifiers.
A health-system-wide count of AI domains visited does not support prioritization. CISOs need to know which models are in use, which departments use them, what data reaches them, and which uses fall outside approved policy. Each of the signal sources below answers only part of that question, so the approach has to be layered.
Build the effort as a cross-functional one. The Health Sector Coordinating Council's Playbook for AI Clinical Resilience, hosted by ASPR TRACIE, describes bringing clinicians, governance leaders, technical experts and compliance officers together to build playbooks for detecting, analyzing and responding to threats in AI-integrated clinical operations. Security cannot decide alone what counts as an approved clinical use.
Detection signal sources and their coverage limits
The limits below are practitioner reasoning about how each control works, not vendor-specific claims. Their value comes from overlap, so plan to correlate at least three of them.
- Secure web gateway and DNS telemetry: Shows which AI domains are requested and from where. Limits: usually shows destinations, not prompt content; misses off-network devices and personal devices; new AI sites need constant catalog updates.
- CASB: Adds application categorization and visibility into sanctioned and unsanctioned cloud services. Limits: coverage depends on how traffic is routed and on how current the application catalog is.
- Endpoint agents: Can see local applications, desktop AI clients and processes. Limits: only managed endpoints report, and clinical workstations with restricted agents may be gaps.
- Identity provider logs: Show sign-ins and OAuth grants to AI services, tied to a named user. Limits: consumer AI sites used with personal accounts never appear.
- Browser extension and API telemetry: Can reveal in-browser AI features, extensions and API calls from internal applications. Limits: requires deployment to managed browsers and does not cover unmanaged paths.
Attributing activity to departments
Detection events carry a device, an IP address or an account. Departments come from joining those to directory data. The usual attributes are the directory department field, job role or title, group membership, and the cost center or service line held in HR systems. Access patterns in the EHR, such as which clinical modules a user works in, can refine this where the directory is stale.
Three accuracy problems recur. Shared workstations in emergency and nursing areas break the link between device and person unless a sign-in event ties the two. Clinicians who float between units or hold several roles may map to more than one department. Directory fields often lag reorganizations. Treat attribution as a confidence-scored mapping, and report departments as aggregates before drilling down to individuals.
There is also a privacy question. Using role and EHR-access attributes to profile staff is itself a data use that privacy, HR and compliance teams should review before it starts. Define who can see user-level results and for what purpose, and keep department-level reporting as the default view for leadership.
Classifying models and AI sites, then measuring
Detection is only useful against a reference list. Maintain an inventory with at least three states: approved for stated uses, approved with conditions, and unapproved. Record each approved use by department and data type, because a model approved for administrative drafting may not be approved for anything involving PHI. Unknown destinations should default to a review queue rather than silently passing as either state.
For each department, track a small set of measures: distinct unapproved AI destinations, active users as a share of department headcount, volume of traffic to those destinations, and the proportion of events involving data types that may contain PHI. Where content inspection is possible, record whether identifiers were present. Where it is not, state that the measure is a proxy.
Pair measurement with policy and training. SAFER high-priority practices call for policies so that patients and clinicians are aware, when possible, that AI is used for clinical or administrative decision making, and for training for clinicians who want to use AI systems that covers known and potential risks. SAFER also states that it is not intended for legal compliance purposes and that following it does not guarantee HIPAA compliance, so treat it as practice guidance, not an audit answer.
Ranking departmental risk
Risk ranking should combine data sensitivity with clinical context. HIPAA-linked risk analysis text published in the Federal Register in January 2025 asks regulated entities to consider the likelihood that a threat occurs and the likelihood that it exploits a vulnerability and results in an adverse effect. Applied here, the threat is PHI reaching an unapproved model, and the likelihood depends on how often a department's workflow involves identifiable data and how easily staff can reach consumer AI.
For clinical context, the FDA's CDRH has said it is considering that action-taking generative AI functions, such as assigning a diagnosis, prescribing a medication or initiating an order set, are typically higher risk than informational ones. That is a consideration, not an adopted rule, but it is a reasonable lens for sorting detected uses. SAFER's examples of AI in patient care, including patient message responses, differential diagnoses, treatment plans and visit notes, give a practical use-type taxonomy.
A workable scoring model has four inputs per department: PHI likelihood in the observed data, use type (informational or action-taking), volume and user count, and the effectiveness of existing controls. Weights are a local decision. Review them with compliance and clinical leadership, and document the rationale, since the ranking itself becomes part of the risk analysis record.
From detection to runtime policy enforcement
Detection and review find a disclosure after it has happened. For PHI, that means the data has already left. HHS reference material on AI risk practices pairs monitoring with safe termination of non-compliant AI, which points to the same idea: observing a problem is not the same as being able to stop it. That material is framed around federal use of AI, so treat it as reference practice for private health systems, not an obligation.
Trussed enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants. For each interaction or tool call, policy can allow, redact, block, rate limit, or escalate for human approval. A request containing identifiers can be redacted before it reaches a model, a request to an unapproved site can be blocked, and a sensitive action can be routed to a person. Enforcement is at runtime, not reviewed after the fact, and the scope is individual interactions and tool calls.
Enforcement also generates the evidence. Each decision produces audit-ready evidence of what policy applied and what was permitted, mapped to HIPAA, including the audit controls under 164.312(b), and to NIST. Trussed can be self-managed in the customer cloud, with configurable retention and data residency. It runs in production with healthcare organizations, and the platform holds SOC 2 Type II, HIPAA, GDPR, FERPA, ISO 27001 and NIST AI RMF credentials. Confirm with counsel how any evidence maps to your audit obligations.
From signal to department-level evidence
Collect
Network, endpoint, identity, CASB and browser signals
Attribute
Map users and devices to departments
Classify
Label each model or AI site approved or unapproved
Rank
Weight PHI exposure and clinical context by department
Enforce
Allow, redact, block or escalate at runtime
Move from review after the fact to enforced policy
Talk through how runtime policy enforcement and audit-ready evidence could apply to your clinical departments.
Request a DemoHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.