LangSmith FINRA Recordkeeping: Where Traces Fall Short as Records

Observability traces and regulatory records have different jobs

A trace answers an engineering question: why did this chain, agent or tool call behave the way it did? It supports debugging, evaluation and iteration. To serve that purpose, observability data is usually designed to be flexible. Retention is configurable, data can be edited or deleted to manage cost and privacy, high-volume applications are often sampled, and nothing in the store proves a record is unchanged.

A regulatory record answers a different question: can the firm show an examiner, years later, exactly what happened and that the evidence has not been altered? FINRA Rule 4511 requires members to make and preserve books and records required by FINRA rules, the Exchange Act and Exchange Act rules. Records with no specified period must be kept at least six years, and FINRA-required records must be preserved in a format and media that complies with SEA Rule 17a-4.

The useful question for a compliance leader is therefore not whether LangSmith is a good tool. It is whether a debugging tool was designed to be the firm's books-and-records system. In most deployments, the honest answer is that it was not, and the gaps below follow from that design intent.

Recordkeeping expectations compared with typical trace store behavior

The table maps each requirement to the characteristic of an observability store that most often creates a gap. Treat the right-hand side as a test to run against your own configuration, not as a statement about any single product tier.

• Retention: Rule 17a-4(a) sets a six-year minimum for certain records, with the first two years easily accessible. Rule 4511 supplies a six-year default. Trace stores typically use a configurable period set for operational reasons, which may be shorter than the period for the record category.
• Non-rewriteable, non-erasable preservation: The WORM requirement prevents alteration, overwriting or erasure. Trace data is typically mutable or deletable by users and administrators.
• Audit-trail alternative: As described by the SEC, the system keeps a complete time-stamped audit trail of modifications and deletions so the original record can be re-created for the retention period. A trace store that allows silent edits or deletion without such a trail does not meet that description.
• Integrity: Examiners expect demonstrable integrity. Trace stores typically offer no cryptographic proof that a trace is unchanged since capture.
• Completeness and production: Rule 17a-4(f)(3)(i) requires facilities for immediately producing preserved records and copies of them. Sampled traces cannot produce interactions that were never captured.

The retention and tamper-evidence gaps that matter in an examination

Retention is the easiest gap to see and often the easiest to fix, but it is rarely just a number. The firm must first classify which AI artifacts are records under Rules 17a-3, 17a-4 or specific FINRA rules, then assign each a retention period. Only then can a trace retention setting be compared with a requirement. A single global retention value cannot reflect different record categories, and it must also be documented and enforced, including how deletion is prevented or captured.

Immutability is the harder gap. Rule 17a-4(f) permits an electronic recordkeeping system, subject to conditions, and the two routes are distinct. WORM prevents change. The audit-trail alternative permits change but makes every modification and deletion recoverable and attributable, with the date and time of operator actions. Neither route is satisfied by a store where an administrator can edit or remove a trace and leave no recoverable history.

Tamper-evidence is how a firm demonstrates the result. Common mechanisms include hash chaining, where each record includes a digest of the previous one so any alteration breaks the chain. Others are append-only storage, object-lock features on cloud storage, signed records, and independent timestamping. They are often implemented in a separate evidence store rather than inside the observability tool. Whichever combination is chosen, the firm must be able to explain it to an examiner. FINRA's cloud report also reminds firms to assess any recordkeeping products or services offered by their cloud providers, so a vendor's assertion does not replace the firm's own assessment.

Finally, Rule 17a-4(f)(2)(v)(A) requires a backup electronic recordkeeping system that serves as a redundant set of records if the original is inaccessible. A single trace store, however well run, is one copy.

Completeness: recorded activity is not the same as enforced policy

Sampling is a sensible cost and performance choice for observability. For a regulatory record it is a completeness gap, because the interactions an examiner asks about may be the ones that were not sampled in.

There is a second, subtler gap. A trace shows what an application did. It does not by itself show what the AI system was permitted to do, or whether a policy was applied at the moment of the call. Supervisory expectations under FINRA Rules 3110 and 3120 call for procedures reasonably designed to comply with recordkeeping obligations, periodically reviewed, with control procedures that test and verify them. Evidence that a rule was enforced is stronger support for that testing than a recording of activity that someone reviews later.

This is where runtime policy enforcement in the execution path is relevant. Trussed enforces policy on individual interactions and tool calls, with actions such as allow, redact, block, rate limit and escalate for human approval. Each enforcement decision produces audit-ready evidence of what the AI was allowed to do and what it did. Trussed supports self-managed deployment in the customer cloud, with configurable retention and data residency. Insurance, a regulated financial sector, is where Trussed is deepest. Any evidence layer, including this one, must still be assessed against Rule 17a-4 by the firm itself.

Criteria for a complementary evidence layer

The practical design is usually two layers with distinct purposes. Observability stays where it is, serving engineers. A separate evidence layer holds what must be preserved, with its own retention, integrity and retrieval controls. The evidence layer should be judged on a short list of criteria.

First, retention must be configurable by record category and documented. Second, preservation must follow either WORM or a complete audit trail, with an integrity mechanism the firm can explain. Third, data residency and deployment model must fit the firm's own control requirements, which is why customer-cloud deployment matters to many compliance teams. Fourth, the retrieval path should be tested separately from the capture path, because immediate production is a condition of its own. Fifth, the layer should evidence enforced policy and not only recorded activity.

The Books and Records Rules apply to all member firms, including where business is conducted through unofficial channels. Shadow or unapproved AI tooling therefore sits inside the same obligation, and the evidence layer needs a defined scope that covers it. Begin with the checklist above, document the gaps, and decide which of them a separate evidence layer should close.