How to Scope AI Agents Into the NYDFS Part 500 Annual Penetration Test

What Part 500 requires, and what it does not say about AI

As of November 2026, the text of 23 NYCRR Part 500 requires each covered entity, in accordance with its risk assessment, to develop and implement written policies and procedures for vulnerability management designed to assess and maintain the effectiveness of its cybersecurity program. It also requires penetration testing of the entity's information systems, from both inside and outside the systems' boundaries, by a qualified internal or external party, at least annually.

There is no AI-specific testing provision. DFS's October 16, 2024 industry letter on AI-related cybersecurity risks states that it imposes no new requirements and helps entities meet existing obligations. It names social engineering, enhanced cyber-attacks, theft of nonpublic information and supply chain dependencies as AI-related risks, and it reminds entities that access privileges must be reviewed at least annually so each user can reach only the nonpublic information their job requires.

The practical consequence is that the scope argument runs through your risk assessment. An agent that reads customer records, calls internal APIs or acts with a user's delegated credentials is an information system component. If it is absent from the test, you need a documented reason tied to risk, not an assumption that the existing application scope covers it.

Step 1: Build an agent inventory that can be tested

Traditional scopes list hosts, applications and network segments. Agents cut across those lists because their reach is defined by what they can call and whose authority they use. Start with one record per agent and enumerate its boundaries, since each one is a discrete thing a tester can probe.

Treat the inventory as a living artifact owned by the security team, with business owners confirming each agent's purpose. An agent missing from the inventory cannot be scoped, and cannot be defended as intentionally excluded.

• Agent and owner: Name, business purpose, accountable owner, and whether it is built in-house or provided by a third party.
• Tools and API calls: Every tool or API the agent can invoke, with the permission level behind each call. Each one is a candidate scope item.
• MCP server connections: Each connected server, who operates it, and what tool descriptions, schemas and return values it supplies to the model.
• Delegated credentials: Service accounts, tokens and user-delegated authority the agent holds, and the privileges behind them.
• Content sources: Documents, emails, web pages and tool outputs the agent ingests. Each is a potential prompt-injection path.
• Data stores and nonpublic information: What the agent can read or write on behalf of users, and where that data is classified.

Step 2: Tie every scope decision to the risk assessment

Because the vulnerability management requirement is framed in accordance with the risk assessment, the strongest scoping record shows a line from each inventory item to a risk assessment entry. If your assessment does not yet mention agent-driven access to nonpublic information, tool permissions or third-party model and MCP dependencies, update it first. Otherwise the test scope will have nothing to trace back to.

For each agent, record an in-scope or out-of-scope decision and the reason. Reasonable criteria include access to nonpublic information, ability to take write or destructive actions, use of delegated credentials, exposure to untrusted content, and reliance on third-party models, tools or MCP servers. Agents that fail none of these tests can be excluded with a stated rationale, which is better than silent omission.

Two boundary questions deserve explicit answers. First, the requirement covers testing from inside and outside the system boundary, so document whether an agent's reach crosses that boundary, for example through an externally hosted model or a third-party MCP server. Second, third-party components raise service provider questions. DFS issued third-party service provider guidance on October 21, 2025, and its AI letter includes a vendor management section, so note which components fall under your existing third-party controls and how testing authorization is handled for them.

The CISO co-signs the annual certification with the highest-ranking executive. The scope rationale should be something you can defend at that level.

Step 4: Write rules of engagement for non-deterministic systems

An agent may respond differently to the same input on consecutive runs, so a single failed attack proves little. Rules of engagement should define how many attempts constitute a fair test for each objective, and how a finding is declared reproducible, for example by a success rate across a stated number of runs. Record the model version, prompts, tool configuration and permissions in effect for each run so a retest can be compared with the original.

Define success by observable action, not by model text. A tool call that moved nonpublic information, a write to a system of record or an action taken with a delegated credential is a finding. A refusal phrased badly is not.

Agree in advance on the environment and data. Decide whether testing runs against production agents or a faithful copy, what synthetic or real data testers may touch, and how nonpublic information encountered during testing is handled. Set stop conditions and a contact path, because an agent with write access can cause real effects. Where a third party operates part of the chain, confirm that testing is authorized for that component before it begins.

Finally, link findings on excessive permissions to the annual access privilege review, so remediation lands in a control cycle that already exists.

Evidence to assemble, and where runtime enforcement helps

No rule prescribes an AI-specific evidence package, so build one that answers the examiner's likely questions: what was in scope, why, how was it tested, what was found and what was done about it. A sound set includes the agent inventory with dated versions, the risk assessment entries each decision traces to, the in-scope and out-of-scope rationale, the test plan and abuse-case matrix, the tester's report with reproducibility notes, remediation tracking, and the access review records for agent entitlements.

One gap remains. A test report shows what testers tried at a point in time. It does not show what agents were permitted to do, or what they did, across the rest of the year. If an agent's permissions are only described in configuration documents, testers have no crisp boundary to test against and examiners have little operating evidence.

This is where runtime policy enforcement supports the testing cycle. Trussed enforces policy in the execution path on individual interactions and tool calls across models, agents, tools and MCP servers: allow, redact, block, rate limit, or escalate for human approval. Because those decisions are enforced at runtime, not reviewed after the fact, they give testers defined control boundaries to attempt to cross, and they generate audit-ready evidence of what agents were allowed to do and what they did, mapped to NYDFS. That evidence sits alongside the penetration test report and supports model risk management conversations. Trussed is not a testing or red-teaming product; it supplies the enforced boundaries and the evidence record around the test.