Detecting a User Probing LLM Guardrails Across Many Sessions

How cross-session probing differs from single-session abuse

Single-session abuse is usually visible inside one conversation: a long jailbreak attempt, a burst of blocked requests, an obvious injection string. A per-prompt filter or per-session counter can catch it.

Cross-session probing is built to avoid that visibility. An adversary submits a few attempts per session, observes which are blocked or altered, refines the wording, and returns later, sometimes under a different session, account, or network address. Each event looks minor. Taken alone, a single blocked prompt is routine, and a single allowed rephrasing looks benign.

OWASP guidance on prompt injection lists persistent manipulation across sessions as an attack pattern, and OWASP's LLM companion guidance names multi-turn trajectory attacks as a distinct pattern to detect. OWASP also notes that trivial changes such as random capitalization, character spacing, or word shuffling can find variants that slip past a filter. That is exactly what iterative probing automates. The problem is therefore a pattern problem across enforcement events, not a prompt-filtering problem.

Signals worth correlating

No single signal is conclusive. The value comes from combining them per identity over a window you choose and tune against your own traffic.

• Near-duplicate prompts: Repeated requests that differ only in spacing, casing, word order, or small substitutions. Literal phrase matching, such as for 'ignore previous instructions', misses these perturbed variants, so use similarity comparison rather than exact strings.
• Policy-block rate per identity: The share of an identity's requests that are blocked or redacted, tracked across sessions rather than within one.
• Incremental rephrasing after a block: A blocked request followed by a close variant that changes only the part the policy objected to.
• Tool-call boundary testing: Repeated attempts to invoke tools, arguments, or resources just outside what the identity is permitted to use.
• Phrasing-family changes: Movement between instruction-override, role-play, and encoding styles aimed at the same blocked outcome.
• Drift in outcomes: A rising share of previously blocked intents now being allowed, which may signal a successful bypass.

Linking sessions to an identity

Correlation is only as good as the keys that connect events. The strongest keys are authenticated: user ID, API key, and tenant. Where authentication is weak or accounts are cheap to create, add client properties such as IP address, User-Agent, and device or certificate attributes.

Web session management guidance from OWASP recommends binding a session to client properties like IP address and User-Agent, and treats a mid-session change as an indicator of manipulation. The same guidance warns that those properties cannot be trusted alone. Apply that caution here. Shared corporate egress addresses, mobile networks, and shared service accounts all create false links, and a determined adversary can rotate IPs and headers to create false separation.

Treat client properties as supporting signals that raise or lower confidence in a link, not as proof of one actor. Record how each link was made so a reviewer can see why two sessions were grouped. Also separate legitimate red-team and security testing by tagging those identities, so authorized testing does not trigger enforcement against your own staff.

Graduated responses and false-positive tradeoffs

Responses should match confidence. Allow ordinary traffic. Redact when a response or input contains content that should not pass, but the intent is not clearly hostile. Rate limit an identity whose block rate and rephrasing pattern are rising, which slows iteration without ending access. Block when the pattern is clear or a tool boundary is repeatedly tested. Escalate for human approval when the action is consequential or the correlation rests on weak identity links.

Allow, redact, block, rate limit, and escalate for human approval are enforced in the execution path at the level of individual interactions and tool calls. Decisions that depend on a cross-session finding, such as tightening limits for a specific identity, take effect once the downstream analysis updates policy.

The tradeoff is familiar. Short windows and strict similarity miss slow probing. Long windows and loose similarity flag shared accounts and persistent but legitimate users. Start with milder responses such as rate limiting and review before hard blocks, and measure outcomes. A metric such as the percentage of adversarial prompts that bypass safety filters, referenced by NIST's AI metrology work, gives you a way to judge whether tuning is improving detection. Regular penetration testing and breach simulation can validate the rules.

Evidence a regulated enterprise should retain

Investigations and audits need to show what was attempted and how policy was enforced. For each decision, retain a timestamp, identity keys, the policy that applied, the outcome (allowed, redacted, blocked, rate limited, escalated), and any approver action on escalations. Retain the correlation findings and the reasoning behind identity links as well.

This has to be balanced against minimization. OWASP guidance says to log only the minimum AI interaction metadata needed for security monitoring and to redact or anonymize prompt and output content before storage. In practice, store metadata and derived features such as similarity scores, with raw text kept minimal and protected. Watch for drift in the event stream so that the evidence remains reliable over time.

NIST AI RMF MANAGE 4 addresses tracking incidents and errors and having processes for response and redress, which bears on how a probing case is documented and closed. Trussed generates continuous audit-ready evidence from enforcement, mapped to HIPAA, NAIC, NYDFS, the EU AI Act, and NIST, with configurable retention and data residency in a self-managed deployment in the customer cloud. Confirm with counsel which obligations apply to your organization.