Testing Broker-Dealer AI Agents with Synthetic Customer Accounts
Build a synthetic account set that represents the personas, entitlements, account states, edge cases and adversarial conditions the agent will meet. Run it in an environment isolated from production data, tools and APIs, with pass/fail criteria and test protocols defined in advance. Then treat the results as bounded: they show behavior under defined conditions, while policy enforcement in the execution path and the evidence it produces cover what the agent is allowed to do and did in live operation.
- Synthetic data store with no production connectivity
- Sandboxed tools and APIs that mimic production interfaces
- Versioned records of agent, model, prompts, data and configuration
- Independent review and documented release approval
- {"title":"1. Translate findings into policy","description":"Turn each failed or borderline scenario into a rule, such as which tools an agent session may call, with what arguments, and for which entitlements."}
- {"title":"2. Choose the action","description":"Assign allow, redact, block, rate limit or escalate for human approval to each rule. Planted-marker leaks suggest redact or block; approval-gated actions suggest escalate."}
- {"title":"3. Enforce in the execution path","description":"Apply the rule to the interaction or tool call as it happens, so enforcement does not depend on the agent following instructions."}
- {"title":"4. Record evidence per decision","description":"Capture which policy applied, what the action was, and what happened, tied to the specific interaction or tool call, with retention and data residency set by the firm."}
- {"title":"5. Feed results back","description":"Use runtime evidence to find gaps, add scenarios to the next synthetic test cycle, and show reviewers the link from test finding to enforced policy."}
Scenario categories and pass/fail criteria
- Data access: the agent returns only data the requesting user is entitled to see. Pass means zero planted markers outside the authorized account.
- Tool-call boundaries: the agent does not call tools or pass arguments outside its documented authority, including when a user asks it to. Pass means no out-of-scope call is executed.
- Suitability-adjacent behavior: when asked for something resembling a recommendation, the agent follows the defined handling, such as declining, providing neutral information or routing to a registered person.
- Communications: drafted customer-facing text is checked against firm content rules, because GenAI-assisted communications must comply with federal securities laws and FINRA rules.
- Escalation: actions that require human approval are routed to a reviewer and not executed without it. Pass means no action completes before approval.
- Adversarial: prompt injection, role-play requests and chained instructions fail to move the agent outside scope. Techniques drawn from adversarial testing and red-teaming practices apply here.
What supervisory material implies for test evidence
Real customer records are a poor test input for an agent that reads account data and calls tools. Privacy, books-and-records and supervisory obligations all complicate their use, so a synthetic account set is the practical alternative. The harder question is what the set must prove, and to whom.
As of November 2026, FINRA states that its rules are technology neutral and continue to apply when member firms use GenAI or similar technologies. Its 2026 Annual Regulatory Oversight Report calls for robust testing to understand a model's capabilities, limitations and performance, and lists privacy, integrity, reliability and accuracy as areas to consider. It also names two emerging agent risks: autonomy, meaning agents acting without human validation and approval, and scope and authority, meaning agents acting beyond the user's actual or intended scope. FINRA notes that agents may call for supervisory processes specific to the type and scope of the agent.
The oversight report also refers to simulating a regulator's examination by requesting records. A useful design test follows from that: could the firm hand an examiner the test protocol, the synthetic data definition, the results and the remediation record on request? FINRA's older guidance on algorithmic trading supports the same habits by analogy: testing before production, independent quality assurance, a segregated environment, and a retained record of protocols, results and remediation of significant defects. Model risk management guidance such as SR 11-7 frames the same work as development, effective validation and governance, with testing continuing after release.
Designing the synthetic account set
Start from the agent's documented scope and authority. Every account attribute you generate should exist to exercise a boundary in that document, not to make the data look realistic. Approval of coverage should be recorded, because reviewers will ask who decided what the set represents.
Generate the data from definitions you control rather than lightly altering real records, so no real customer information can be recovered from the test set. Keep a versioned definition of each persona so a run can be reproduced later.
- Customer personas: Represent the account holder types the agent serves, such as individual, joint, trust, and retirement accounts, plus the internal roles that use it, such as service staff and supervisors.
- Entitlements: Vary what each user and each agent session may see and do, including users with narrow access and users whose access has just been removed.
- Account states: Include restricted, dormant, newly opened, recently transferred and flagged accounts, since agents often behave differently at state boundaries.
- Data edge cases: Add missing fields, conflicting records, stale values and unusual formats to test integrity and accuracy.
- Planted markers: Seed identifiable fake values in sensitive fields so any leakage into a response, tool call or log is detectable.
- Refresh cadence: Plan new synthetic datasets and stressed scenarios for each test cycle, so retesting does not repeat only what the agent has already seen.
What pre-production testing cannot establish
A passing result shows that the agent behaved acceptably under the conditions you defined. It does not show how it will behave under conditions you did not anticipate, with real account diversity, or after changes to models, prompts, tools or data. Model risk management practice reflects this by treating ongoing testing and monitoring as part of the lifecycle, not an afterthought to validation.
It also does not discharge supervision, communications or recordkeeping obligations. Because those obligations apply to agent behavior regardless of the technology, passing tests is evidence of diligence, not of compliance in operation.
The agent risks FINRA describes, acting beyond intended scope and acting without human approval, are about live behavior. A test can sample those behaviors, but relying only on pre-release results leaves a gap between what was tested and what the agent can do. Closing that gap means limiting actions at the moment they occur.
From synthetic test to runtime evidence
Scope
Document the agent's intended scope and authority as the baseline.
Synthetic accounts
Personas, entitlements, states, edge cases and adversarial scenarios.
Isolated test
Sandboxed data, tools and APIs with pre-defined pass/fail criteria.
Release gate
Independent review and approval by release authorities.
Runtime enforcement
Allow, redact, block or escalate each interaction and tool call, with evidence.
Connect test findings to enforced policy and evidence
See how policies derived from pre-production testing can be enforced on interactions and tool calls, and how that enforcement produces audit-ready evidence.
See How Evidence Is ProducedHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.