AI Governance Software Buyer's Guide: Questions to Ask Before You Buy

Runtime Enforcement vs. Design-Time Governance

Most AI governance software marketed to enterprises documents policy rather than enforcing it. NIST's AI Risk Management Framework (AI RMF 1.0) organizes AI risk management into four functions, Govern, Map, Measure, and Manage, and separates pre-deployment risk identification from Manage, which addresses ongoing risk treatment and monitoring after deployment. That separation is a useful lens for buyers. A platform that produces model cards, risk assessments, or policy documents is doing Map-function work. A platform that can intercept an agent's action, evaluate it against policy, and allow, deny, or modify it before execution is doing Manage-function work, in real time.

The distinction matters because most operational failures in AI agent deployments occur after design-time review is complete. An agent granted a tool integration, an API credential, or a data access scope during development can behave differently in production as inputs, tool availability, or task context change. Design-time governance cannot observe or constrain that behavior. Runtime governance can. When evaluating vendors, ask whether the platform's core mechanism is documentation and reporting, or active mediation of agent actions during execution. The answer determines whether the product addresses the risk enterprises are actually trying to manage.

Agent Identity and Least-Privilege Permissions

OWASP's Top 10 for LLM Applications identifies "Excessive Agency" as a risk category describing systems granted more functionality, permissions, or autonomy than their task requires. Governing that risk requires knowing which agent performed which action, under which permission scope, at which point in time.

Buyers should ask whether the platform issues distinct, auditable credentials per agent or per session, rather than relying on shared service accounts or static API keys. Shared credentials make it difficult to attribute an action to a specific agent, revoke access for a single compromised or retired agent without affecting others, or enforce least privilege at a granular level.

A related operational question concerns lifecycle management: how are permissions assigned when an agent is added, modified in scope, or decommissioned, and how quickly can access be revoked. A governance platform that cannot answer these questions with technical specificity, rather than general policy language, is unlikely to support least-privilege enforcement in practice.

MCP Security and Tool-Call Oversight

The Model Context Protocol, introduced by Anthropic in November 2024, defines an open, client-server architecture through which AI models discover and invoke external tools and data sources. MCP's specification addresses trust boundaries between hosts, clients, and servers, and recommends mechanisms for user consent before a tool or resource is invoked.

Because MCP's client-server model creates identifiable points where a request passes between an agent and a tool, it also creates identifiable points where governance controls can inspect or block that request before execution. Buyers should ask at what layer, client, gateway, or server, a vendor's policy enforcement is applied, and whether enforcement requires custom instrumentation for each tool integration or works generically across MCP-compliant servers.

Equally important is payload-level visibility. OWASP's guidance on insecure plugin and tool integration design underscores that oversight limited to aggregate usage metrics, such as call counts, does not reveal whether a specific request or response violated policy. Ask whether the platform can inspect full request and response payloads exchanged between an agent and a tool, not just summary statistics.

Auditability and Logging Requirements

The EU AI Act, which entered into force in 2024, includes record-keeping obligations under Article 12 requiring high-risk AI systems to enable automatic logging of events over the system's lifetime, supporting traceability and post-market monitoring. Whether or not a given deployment falls under the Act's high-risk classification, the underlying requirement, that agent actions are logged in a form suitable for later audit, reflects a broader compliance expectation.

Buyers should confirm what is captured in each log entry: the request made, the response received, and the policy decision applied, not merely a timestamp and an agent name. Logs should also be assessed for tamper-evidence and access control, since a log that can be altered after the fact provides limited assurance to internal risk and compliance functions.

Ask how long logs are retained, whether retention is configurable to match internal or regulatory requirements, and whether logs can be exported in a format usable by existing audit tooling. A platform that treats logging as an afterthought, rather than a structured record tied to each governed action, will not satisfy most enterprise audit functions.

Core Questions to Ask Every Vendor

• Can the platform intercept and block a non-compliant agent action in real time, demonstrated in a live environment rather than a static architecture diagram?
• How does the platform assign and verify unique agent identities, and how are least-privilege permissions enforced and revoked?
• Does the platform support Model Context Protocol integration, and at what layer (client, gateway, or server) is policy enforcement applied to tool calls?
• What logging is captured for each agent action, including request, response, and decision outcome, and can these logs satisfy internal audit and regulatory record-keeping requirements?
• What is the platform's failure behavior, fail-open or fail-closed, if the enforcement point becomes unavailable?

Evaluating Vendor Claims

No primary framework reviewed for this guide, including the NIST AI RMF, the MCP specification, OWASP guidance, or the EU AI Act, defines a formal certification standard specifically validating "runtime AI governance" as a product category. Alignment with these frameworks supports a defensible internal governance process, but framework alignment alone does not confirm that a vendor's enforcement mechanism functions as described.

What to Verify Before You Buy