How to Audit an LLM Application: A Step-by-Step Security Guide

What an LLM application security audit should cover

An effective audit evaluates the application path around the model. Prompts and model configuration matter, but enterprise risk usually appears where user input, retrieved content, permissions, tools, identities, and runtime decisions interact.

Prompt review is necessary, but it is not sufficient. In enterprise LLM applications, the prompt is assembled dynamically from user input, system instructions, retrieved context, policy metadata, tool outputs, conversation history, and identity context. Any of these inputs can affect model behavior. Retrieved documents, webpages, emails, tickets, uploads, and tool outputs should be treated as untrusted inputs because they can carry indirect prompt injection instructions.

For RAG systems, auditors should review ingestion controls, document provenance, data classification, chunking and embedding processes, retrieval filters, authorization checks, and whether users can retrieve content they are not entitled to access. The audit should test whether malicious or irrelevant retrieved content can override system intent, exfiltrate context, or cause the model to produce unsafe tool calls.

For AI agents, the most important audit boundary is the action layer. Excessive agency often comes from excessive functionality, excessive permissions, or excessive autonomy. An agent that can read broad data sources, call many tools, and execute actions without approval presents a different risk profile than an assistant limited to summarization.

Step-by-step audit workflow

This workflow keeps the assessment practical: first establish context and exposure, then map the application, test LLM-specific failure modes, review deterministic controls, and document remediation evidence.

1. Define context and exposure

   Document the application purpose, users, deployment environments, data classes, model actions, prohibited actions, regulatory constraints, and whether the system is exposed to untrusted users or untrusted content.

2. Build the architecture and data-flow inventory

   Map entry points, identity bindings, prompt construction, retrieval paths, model calls, tool calls, storage, logs, external integrations, secrets, and administrative interfaces.

3. Threat model LLM-specific risks

   Assess prompt injection, sensitive information disclosure, supply-chain exposure, data or model poisoning, improper output handling, excessive agency, system prompt leakage, vector weaknesses, misinformation, and unbounded consumption.

4. Review controls and permissions

   Verify least privilege for tools, APIs, agents, service accounts, data sources, retrieval filters, runtime policies, approval gates, monitoring, and incident response processes.

5. Run adversarial tests

   Test direct prompts, indirect prompt injection through retrieved content, multi-turn bypass attempts, unsafe tool use, data leakage, retrieval manipulation, agent loops, and resource abuse.

6. Prioritize and evidence findings

   Assign severity using exploitability, data sensitivity, business impact, tool authority, blast radius, runtime exposure, monitoring coverage, and compensating controls.

How to test LLM-specific risks

LLM testing should include both direct interaction with the application and indirect interaction through the content that the application retrieves or consumes. The goal is to determine whether untrusted input can alter model behavior, expose sensitive context, or trigger unauthorized actions.

Auditors should treat retrieved documents, webpages, emails, tickets, uploads, and tool outputs as untrusted inputs. These sources can carry instructions that are not visible in the primary user prompt but can still influence model behavior once included in context.

Audit evidence to collect

Audit evidence should make findings reproducible and remediation decisions defensible. Evidence should show how the application is constructed, where sensitive data moves, what permissions exist, how controls are enforced, and how the system behaves under adversarial testing.

• Application purpose, intended users, deployment environments, data classes, model actions, and prohibited actions.
• Architecture and data-flow inventory covering entry points, prompt construction, retrieval paths, model calls, tool calls, storage, logs, integrations, secrets, and administrative interfaces.
• Identity bindings, service accounts, agent credentials, tool permissions, data-source access, and retrieval filters.
• RAG evidence, including ingestion controls, document provenance, data classification, chunking and embedding processes, authorization checks, and vector store access patterns.
• Runtime policy controls before model calls, after model responses, and immediately before tool execution.
• Adversarial test results for direct prompts, indirect prompt injection, multi-turn bypass attempts, unsafe tool use, data leakage, retrieval manipulation, agent loops, and resource abuse.
• Monitoring coverage, incident response processes, approval gates, transaction limits, and compensating controls.

Prioritizing remediation after the audit

Findings should be prioritized based on the practical likelihood and impact of exploitation. LLM application severity should account for more than prompt behavior. It should consider the authority of connected tools, the sensitivity of available data, the blast radius of a successful action, and whether runtime controls can detect or stop the issue.

Severity should reflect exploitability, data sensitivity, business impact, tool authority, blast radius, runtime exposure, monitoring coverage, and compensating controls.

Issues involving excessive agency often require prompt changes, but they also require deterministic control changes. If an agent can call many tools, read broad data sources, or execute actions without approval, remediation should reduce functionality, reduce permissions, add approval gates, constrain parameters, or separate identities by task.

Where runtime governance fits

Runtime policy checks should be placed where they can actually stop unsafe behavior: before the model call when input or context is disallowed, after the model response when output violates policy, and immediately before tool execution when an action would exceed permissions or business rules. This creates a control path that does not depend solely on the model obeying instructions.