Implementation Guide

    Producing Medicare Advantage Program Audit Universes from AI Agent Logs

    Treat the audit universe as a data-engineering and evidence-integrity problem. Scope which agent activity falls under each audit protocol, define an evidence schema from the protocol's universe layout, map captured fields to that layout, run completeness and accuracy checks against source counts, and document every transformation. Evidence captured at policy enforcement in the execution path is a more reliable source than records reconstructed from general-purpose logs.

    A repeatable approach in six steps

    1. 1

      1. Scope

      Determine which agent interactions and tool calls fall within each applicable protocol and audit timeframe.

    2. 2

      2. Define the evidence schema

      Derive required fields from the protocol's universe layout, not from what existing logs happen to contain.

    3. 3

      3. Capture

      Record evidence for each in-scope event in a store separate from general application logs.

    4. 4

      4. Map and extract

      Apply versioned transformation logic to populate universe fields from captured records.

    5. 5

      5. Validate and reconcile

      Run completeness, accuracy, deduplication and timestamp checks, and resolve exceptions.

    6. 6

      6. Attest and retain

      Obtain owner sign-off and keep derivation documentation with the submitted universe.

    Readiness checks before a universe request arrives

    • The protocol year, program areas and timeframes in scope are recorded for each contract.
    • The evidence schema is versioned and traceable to universe field definitions.
    • Record identity rules are written and tested for duplicates.
    • Timestamp normalization is documented, including the time zone basis.
    • Source-to-universe record counts are reconciled and differences explained.
    • Transformation logic, approvals and reconciliation outputs are retained with each submission.

    Why agent records are hard to turn into a universe

    CMS conducts Medicare Part C and Part D program audits using protocols and program-area data request documents that are posted each year. Sponsoring organizations are asked to submit universes, which are structured lists of records for a given audit area, and the audit then samples from them. As of November 2026, the current-year protocol for each in-scope program area is the authoritative definition of what a universe must contain.

    When an AI agent takes part in an audited process, the record of what it did usually sits in application logs, model gateway records and tool-call traces. Each system uses its own identifiers, timestamp formats and event vocabulary. None was designed around a universe field definition, an audit timeframe or a completeness expectation. Reconciling them after an audit request arrives is slow and difficult to defend.

    CMS has stated, in a Part D pharmacy benefit manager context, that reporting data is not the same as reporting sufficient information to support an audit. A universe file alone is therefore not the deliverable. The deliverable is the universe plus the means to show where each record came from and how it was derived.

    Scoping and schema design

    Start from the current-year protocol and data request document for each program area in scope. CMS generally audits at the parent organization level and applies the program-area protocols that fit the organization type. For example, it would not apply the ODAG protocol to an organization offering only a standalone prescription drug plan. Scoping should record which contracts, protocols and timeframes apply, and which agent workflows touch each one.

    Protocols change between audit years. CMS has revised how Part C universes for ODAG are collected, and it has signalled adding specific documentary evidence requirements to other protocols, such as committee member lists and meeting minutes for SNP Care Coordination. The schema should therefore be treated as a versioned artifact tied to a protocol year, with a controlled process for updating it.

    The schema itself should name the event types that matter, such as a user or system request, a model response, a tool call and its outcome, and a human escalation and its decision. For each, define the fields needed to populate the universe: a stable record identifier, a normalized timestamp with its time zone basis, the acting agent and the contract or plan context, the policy outcome, and a reference to the source event. Record identity needs an explicit rule. CMS has required elsewhere that each record eligible for sampling be uniquely defined, and the same principle applies here. Decide up front whether a record is an interaction, a tool call or a decision, and apply that choice consistently.

    Mapping, validation and the usual failure points

    Mapping converts captured fields into universe fields. Every conversion, such as renaming an event type, converting a time zone or deriving a status from a tool-call result, should be written down as transformation logic that is versioned and approved. Undocumented transformations are among the hardest things to defend, because the auditor cannot tell whether a value is a source fact or an analyst's judgment.

    Validation should cover both completeness and accuracy. For completeness, tie the count of in-scope source events to the count of universe records, and explain every difference, whether from deduplication, exclusion by scope rule or extraction failure. For accuracy, sample universe records back to their source events and confirm that each field matches. Retain the outputs, not just the pass result.

    Four failure points recur. Missing records occur when a system drops events or an extraction window misses late-arriving data. Inconsistent timestamps occur when systems use different clocks, formats or time zones, so records fall on the wrong side of a timeframe boundary. Ambiguous record identity produces duplicates or merged records when the same activity appears in more than one log. Undocumented transformations leave no way to reproduce the universe. Each can be tested for, but only if the checks are designed before the extraction is run.

    Evidence at enforcement versus reconstruction from logs

    Reconstruction starts from records that were created for operational purposes and tries to infer audit facts from them. Gaps appear where a field was never recorded, where a policy decision was applied but not logged in a consistent form, or where retention differs across systems. Completeness is hard to prove because there is no authoritative count of what should exist.

    Capturing evidence where policy is enforced changes the starting point. Trussed enforces policy in the execution path of individual interactions and tool calls, with outcomes such as allow, redact, block or escalate for human approval. Because the decision is enforced at runtime, not reviewed after the fact, the evidence of what was permitted and what happened is produced as part of the same action. That gives a defined source of record for the universe and a basis for source-to-universe tie-outs. Trussed generates audit-ready evidence mapped to frameworks including HIPAA and NIST, and supports self-managed deployment in the customer cloud with configurable retention and data residency.

    The scope matters. Enforcement evidence covers individual interactions and tool calls. If a protocol requires relationships across a longer workflow, the universe design has to account for that separately. The same validation, reconciliation and documentation steps still apply to enforcement-sourced data.

    From agent activity to a defensible universe

    Scope

    Identify agent activity that falls within each audit protocol.

    Capture

    Define and record the evidence schema at the point of enforcement.

    Map

    Translate captured fields to the specified universe layout.

    Validate

    Tie source counts to universe counts and test accuracy.

    Document

    Retain derivation logic and reconciliation outputs for auditors.

    Build the universe from evidence, not reconstruction

    Talk through how audit-ready evidence from enforcement in the execution path of agent interactions and tool calls can feed your universe production process.

    Request a Demo
    Next step

    How does your AI governance program compare?

    See where your program has gaps in less than 2 minutes.