LLM Governance Tools That Log Tool Calls, Not Just Prompts: Evaluation Criteria

What a complete tool-call record contains

Tool calls are where an AI system acts on data and systems, whether through an API, an MCP server or a coding assistant. A prompt log cannot reconstruct those actions, so the tool-call record needs to stand on its own.

OWASP's agent security guidance gives a workable baseline. An approval record should carry the actor, the tool name, the target resource, normalized parameters, a timestamp and an expiry. Add the outcome of the call, since the same guidance recommends logging all agent decisions, tool calls and outcomes. In practice that gives you a record with these fields: caller identity, tool or MCP server identity, target resource, normalized arguments, policy or approval decision, outcome and timestamp.

Two details matter more than they first appear. First, a bare "user_confirmed" flag is not enough. The approval must belong to the current actor and the exact tool call, still be valid, and not have been consumed already. Second, for agents embedded in retrieval pipelines, OWASP recommends traceability from the query that triggered a retrieval, to the output it influenced, to the tool call that output triggered. Ask whether a tool can link a call back to its cause, not just record that it happened.

How the main tool categories handle tool calls

The market overlaps, and vendor descriptions often span categories, so classify a product by what it does with tool calls rather than by its label. The table gives the typical emphasis of each category and the question to put to a vendor. Treat it as a starting point for testing, not as a verdict on any product.

• LLM gateways: Centered on routing LLM traffic, input and output controls, centralized logging and usage visibility. Verify whether tool calls are recorded as first-class events with arguments and outcomes, or only appear inside model traffic.
• Observability platforms: Centered on traces, metrics and debugging. Verify what instrumentation is needed to capture tool calls, whether coverage includes MCP servers and coding assistants, and whether the platform can block anything or only report.
• Governance and GRC platforms: Centered on inventories, policies, assessments and attestations. Verify whether they ingest call-level records from running systems or depend on manual documentation.
• Runtime enforcement: Sits in the path of the interaction and applies policy to each call before it executes. Verify the available decisions, failure behavior, and what record each decision produces.

Observing a tool call is not the same as controlling it

A passive logger records a call after the fact. It can support investigation, but it cannot stop a harmful call. An enforcement layer evaluates the proposed call first and returns a decision: allow, redact, block, rate limit or escalate for human approval. The record then includes the decision, not just the event.

OWASP guidance places enforcement at the tool boundary, independent of the model. The model deciding to call a tool is not the same as the user being authorized to use it, so tool-level authorization, allowlists of permitted actions and parameters, and per-tool permission scoping should not depend on model judgment. Model-based screening of proposed calls against user intent can add a probabilistic layer, but it sits alongside deterministic controls and does not replace them.

Enforcement has costs. An in-path control adds a dependency to every call, so its latency and its behavior when it fails are part of the evaluation. Passive approaches avoid that dependency but accept that policy is reviewed after the fact. No latency or coverage figures should be taken on trust. Measure them in a pilot against your own tool traffic, including MCP servers and coding assistants, which are easy to leave outside the monitored path.

Runtime limits belong in this layer too. Circuit breakers, recursion and retry limits, and chain-depth caps address tool-call loops and spikes, and monitoring for abnormal invocation frequency or repeated approval bypass attempts gives security teams something to alert on.

Records that hold up as audit evidence

A log is not evidence merely because it exists. Evidence has to show what the system was permitted to do and what it did, and a reviewer has to be able to trust it. That points to decision records (policies applied, approvals granted, outcomes) in addition to raw call logs.

The most important design test is failure behavior. OWASP recommends failing closed when risk classification, approval validation, policy lookup or audit logging fails, and for unknown tools or missing approval requirements. A tool that keeps executing calls when its own logging has broken cannot support a claim of complete records. Ask vendors how this is configured and how you can test it.

There is also a real tension between completeness and data minimization. Guidance says to log security-relevant metadata and decisions while keeping credentials, secrets and unnecessary sensitive content out of logs, and to redact before storage. Check how a tool redacts arguments, whether the redaction itself is recorded, and whether the stored record is still useful for investigation afterward.

Finally, confirm retention, integrity protection and export against your own regulatory and internal requirements. Claims of "full audit trails" or compliance packs in vendor descriptions are claims, not proof. Ask to see a sample record for a blocked call and for an approved call, and judge whether an auditor could follow it unaided.

Where runtime enforcement in the execution path fits

Runtime enforcement is one layer, not a replacement for the others. Static scanning of tool code, design-time risk assessment and governance inventories address different problems. Enforcement answers a narrower question: what happens to this specific call, right now.

Trussed operates in that layer. It enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants, with decisions to allow, redact, block, rate limit or escalate for human approval. The scope is individual interactions and tool calls. Because decisions are enforced at runtime, not reviewed after the fact, the enforcement itself generates continuous audit-ready evidence, mapped to HIPAA, NAIC, NYDFS, the EU AI Act and NIST. Deployment is self-managed in the customer cloud, with configurable retention and data residency.

Trussed has run continuously in production in a regulated enterprise for more than 18 months, at 99.95% uptime on live production traffic, with more than 100 models under one policy plane and over 30 GB of audit evidence generated per day. Those figures are relevant to the latency and failure-mode questions above, but you should still test any in-path control against your own workload.