Building an AI Center of Excellence: Governance Structure and RACI Model
An AI Center of Excellence governance structure separates policy-setting authority, held by a central governance body, from execution responsibility, held by security, engineering, and legal teams. A RACI model formalizes this by assigning a single Accountable role for each governance function, including agent permissions, tool-call oversight, and audit logging, while distributing Responsible and Consulted roles across the teams that implement and operate AI systems.
Core Functions an AI CoE Must Govern
Before assigning accountability, a CoE charter should identify the discrete governance functions that require ownership. Four functions are foundational to any enterprise AI program.
Policy Enforcement
Defines who can set and change AI use policy across the enterprise.
Model Approval
Pre-deployment risk classification and sign-off before production release.
Agent Permissions
Least-privilege scoping of tool access and identity for autonomous agents.
Runtime Oversight
Continuous monitoring and enforcement of agent behavior after deployment.
Why AI Governance Requires a Defined Organizational Structure
Enterprises deploying AI at scale often distribute governance responsibility across security, engineering, legal, and business teams without a single structure to coordinate decisions. This produces gaps in ownership: no team can reliably confirm who approved a model for production, who is accountable when an AI agent exceeds its permitted tool access, or who retains audit evidence for a compliance review.
An AI Center of Excellence addresses this by centralizing governance functions, policy definition, and risk acceptance, while distributing execution responsibility across the teams that implement and operate AI systems.
Regulatory context: NIST's AI Risk Management Framework (January 2023), built around four functions (Govern, Map, Measure, and Manage), explicitly calls for organizations to establish defined roles, responsibilities, and accountability structures for AI risk decisions across the lifecycle. ISO/IEC 42001 (December 2023) similarly requires documented organizational roles as part of an AI management system. Neither standard prescribes a specific organizational chart; both leave structure to the enterprise.
A formal CoE paired with a working RACI model, rather than a generic reporting line, is the practical mechanism for closing accountability gaps.
Identifying the Governance Functions That Require Ownership
Drawing on NIST's Govern and Map functions and OWASP's guidance on large language model risk, five functions recur across enterprise AI programs:
- Policy definition and enforcement
- Model risk classification and approval
- Agent identity and least-privilege permissioning
- Tool-call and runtime oversight
- Audit logging with traceability
These functions are organizationally distinct. Policy definition and model approval are typically pre-deployment activities suited to a standing review board. Agent permissioning and tool-call oversight are runtime concerns, since an agent's tool access can change dynamically once it is operating in production.
OWASP's Top 10 for LLM Applications identifies Excessive Agency as a distinct risk category, requiring dedicated constraints on autonomous permissions and tool scope. Audit logging and traceability sit apart from both: the role responsible for retaining and producing audit evidence should not be the same role that approved the agent's initial deployment. Treating all of these as a single approval gate makes it difficult to answer basic accountability questions once agents are running with tool access in production.
RACI Model for AI CoE Governance Functions
The table below maps each governance function to an Accountable role (the single owner) and the Responsible, Consulted, and Informed parties. Each row represents a distinct governance decision point.
| Governance Function | Accountable | Responsible | Consulted | Informed |
|---|---|---|---|---|
| Policy Definition and Enforcement | AI CoE Director | Legal, Compliance | Security, Engineering leads | All business units |
| Model Risk Classification | AI CoE Director | Risk and Compliance team | Legal, Engineering | Product owners |
| Pre-Deployment Model Approval | AI Governance Review Board | Engineering, Security | Legal, Risk | Business unit sponsors |
| Agent Identity and Permissions | Security Engineering lead | Platform Engineering | AI CoE, Legal | Product owners, Compliance |
| Tool-Call and Runtime Oversight | Security Operations lead | Platform Engineering, SecOps | AI CoE, Compliance | Engineering teams |
| Audit Logging and Traceability | Compliance lead | Platform Engineering | Security, Legal | AI CoE, Business units |
RACI rule: Each governance function should have exactly one Accountable role. If a review finds two teams both claiming accountability for agent permissions or audit logging, that is a structural gap, not a sign of strong coverage.
Runtime Oversight of AI Agents: Permissions and Tool-Call Governance
Runtime governance is structurally different from pre-deployment approval. A model approval process runs once before production. Runtime oversight must run continuously, because an autonomous agent's actual behavior, including the tools it calls, the data it reads, and the actions it takes, can diverge from its approved scope after deployment.
Least-Privilege Permissioning
Agent identity and tool access should be scoped to the minimum required for each task. Overly broad permissions create the conditions for Excessive Agency risk, where an agent takes consequential actions outside the boundaries originally assessed during approval. Permissioning decisions should be logged and attributable to the Security Engineering lead accountable for that function.
Tool-Call Oversight
Every tool call an AI agent makes should be observable, logged, and reviewable. Security Operations holds accountability for this function, which means SecOps must have the instrumentation to detect and flag tool calls that deviate from expected patterns. This requires coordination with Platform Engineering, which is responsible for implementing the logging infrastructure.
Audit Log Independence
The Compliance lead accountable for audit logging should not also be accountable for model approval or agent permissioning. Separating these roles ensures that audit evidence can be produced independently of the teams whose decisions are under review.
Implementation Considerations for Building the RACI Model
Several practical issues arise when translating a RACI model from a document into an operating governance structure.
Start with functions, not titles
Map governance functions before assigning named individuals. Organizations that assign named owners first often discover later that a single person holds accountability for functions that should be separated, or that a function was never assigned because no existing role seemed to fit.
Distinguish pre-deployment from runtime accountability
Pre-deployment and runtime governance have different operational rhythms. Model approval is a periodic gate; tool-call oversight is a continuous operational function. Assigning both to the same role or team without distinguishing them produces a structure that handles approval gates reasonably but lacks the capacity to monitor agents in production.
Formalize escalation paths
The RACI model should specify what happens when a Responsible team identifies a compliance issue during runtime oversight. The escalation path from Security Operations to the AI CoE Director, or from Platform Engineering to the Compliance lead, should be documented and tested before agents reach production.
Align the RACI model to existing risk frameworks
Organizations already operating under NIST CSF, ISO 27001, or SOC 2 can anchor AI governance accountability to existing risk management structures. This reduces the overhead of maintaining a parallel governance process and makes it easier to produce evidence for auditors who are already familiar with existing control frameworks.
Revisit role assignments as AI programs scale
A RACI model that works for three deployed agents may not scale to thirty. As the number of agents and tool integrations grows, the runtime oversight function in particular may require dedicated staffing rather than a secondary responsibility held by a Security Operations team primarily focused on traditional infrastructure.
Frequently Asked Questions
Does every organization need a formal AI Center of Excellence?
Not every organization needs a standalone CoE as a separate business unit. What every organization deploying AI at scale does need is a defined structure that assigns accountability for policy, model approval, agent permissions, and audit logging. Whether that structure is called a CoE, a governance board, or an AI risk committee is less important than whether accountability is documented, assigned to specific roles, and enforced in practice.
How does a RACI model help with AI agent governance specifically?
AI agents introduce governance challenges that traditional software approval processes do not fully address, because agents operate autonomously after deployment and their behavior can change based on context. A RACI model forces organizations to assign explicit accountability for runtime functions, including tool-call oversight and agent permissioning, that often fall between existing team boundaries. Without a RACI model, it is common to find that security owns permissioning during deployment but no team owns it during operation.
What is the difference between Accountable and Responsible in this context?
Accountable means the single role that owns the outcome and answers for it in a review or audit. Responsible means the team or teams that do the work to fulfill the function. For model approval, the AI Governance Review Board is Accountable because it signs off on the decision; Engineering and Security are Responsible because they produce the risk assessment and technical evidence the board reviews. A function can have multiple Responsible parties but should have only one Accountable role.
How should organizations handle governance when AI is deployed by multiple business units independently?
Federated AI deployment without central policy authority is one of the most common sources of governance gaps. The CoE governance structure should define minimum standards that apply across all business units, including required approval gates and logging requirements, while allowing business units to operate their own implementation processes within those standards. The CoE Director or equivalent role holds accountability for policy definition even when individual business units hold responsibility for implementation.
Which standards or frameworks are most relevant to AI CoE governance?
NIST's AI Risk Management Framework provides the most widely referenced structure for AI governance roles and accountability, organized around the Govern, Map, Measure, and Manage functions. ISO/IEC 42001 provides a management system standard that includes documented organizational roles as a formal requirement. OWASP's Top 10 for LLM Applications is the most practically useful reference for understanding the specific risks, including Excessive Agency, that runtime governance must address. Organizations subject to sector-specific regulations should also map their AI governance structure to applicable requirements in those frameworks.
Formalize Governance Before Agents Reach Production
A defined CoE structure and RACI model reduce ambiguity over who approves AI agent permissions, enforces policy, and produces audit evidence. Runtime governance extends that structure into execution, where agent behavior and tool access require continuous oversight.
Learn About AI Agent Security