How to Evaluate an AI Agent Marketplace Listing Before Installing

Why Marketplace Listings Require Technical Scrutiny

AI agent marketplaces, public and internal, do not follow a standardized manifest format. A listing's description, screenshots, and category tags are marketing text, not a technical specification of what the agent will do once installed. The artifact that matters is the underlying manifest or tool schema: the set of tool names, input schemas, and permission scopes the agent declares. OWASP's Top 10 for LLM Applications identifies the resulting risk categories directly. Excessive Agency describes a system granted more functionality, permissions, or autonomy than its operational purpose requires. Supply Chain risk covers unverified third-party components, plugins, and models entering the environment without review. A marketplace listing that only advertises capability in prose, without an inspectable manifest, cannot be evaluated against either risk category with any confidence.

Reviewing MCP Configuration and Authorization Scope

The Model Context Protocol defines a client-host-server architecture in which servers expose tools, resources, and prompts that a host application invokes on behalf of a model. Tool definitions include a name, description, and JSON Schema-based input schema, which is what a client uses to determine what parameters and calls a tool actually permits, independent of how the listing describes it. For HTTP-based transports, MCP includes an authorization framework built on OAuth 2.1 concepts. This matters for evaluation because scope negotiation happens during the client-server initialization handshake, not solely through the listing description. Critically, MCP does not itself guarantee uniform enforcement of these scopes; enforcement depends on how the host or client implementation applies them at runtime. A listing that references MCP support should be evaluated on where and how authorization is enforced, not on the presence of the term alone.

Governance Alignment: Mapping Permissions to Business Justification

NIST's AI Risk Management Framework organizes risk management into four functions: Govern, Map, Measure, and Manage. The Manage function is directly applicable here: each declared permission or tool should be mapped to a specific, documented business justification before approval, rather than approved as a bundle. Marketplace agents should be treated as untrusted third-party code, subject to the same supply-chain scrutiny as any external software dependency, consistent with OWASP's Supply Chain and Excessive Agency categories. This is complicated by the fact that no universal manifest standard exists across AI agent marketplaces today, so governance review cannot assume that self-reported permissions are complete or accurate. SBOM-style provenance disclosure, now standard practice in traditional software supply-chain governance under frameworks like NIST SP 800-218, is not yet consistently required for AI agent listings, which means provenance verification often falls to the reviewing team rather than the marketplace itself.