Article 15 as a Technical Control Specification
Article 15 of Regulation (EU) 2024/1689 requires high-risk AI systems to achieve an appropriate level of accuracy, robustness, and cybersecurity, and to maintain that performance consistently throughout the system's lifecycle. The provision is not limited to a one-time design check. Article 15(2) requires providers to declare accuracy levels and relevant metrics in the instructions for use, creating a binding disclosure that must match internal test evidence. Article 15(3) requires resilience against errors, faults, or inconsistencies, particularly those arising from interaction with people or other systems, and mandates technical redundancy such as backup or fail-safe plans. It also addresses systems that continue learning after deployment, requiring measures to reduce the risk of biased outputs feeding back into future inputs. Article 15(4) requires resilience against unauthorized attempts to alter system use, outputs, or performance, naming data poisoning, adversarial examples, and confidentiality attacks as specific vulnerability categories. None of this text was written with tool-calling agents in mind, but the obligations apply regardless. Autonomous agents that call external tools, retrieve data, or retain memory across sessions still fall within the scope of accuracy, robustness, and cybersecurity requirements, and the audit burden falls on demonstrating that guardrails address each clause with concrete technical controls.
Why This Differs From Static Model Compliance
Most Article 15 commentary assumes a single trained model producing a single output. An agent with tool-calling capability introduces additional surface area: each tool invocation is a potential point of unauthorized alteration, each retrieved document is a potential poisoning vector, and each multi-step action chain compounds the risk that an early error propagates unchecked. An audit methodology built only around model-level accuracy metrics will miss these points. Evidence must extend to the permission boundary around each tool call, the verification step applied to tool outputs before they influence subsequent agent actions, and the logging that lets an auditor reconstruct what the agent was authorized to do versus what it attempted.
Guardrail Testing Methodology
- Define accuracy metrics at the task-outcome level before deployment, not retroactively, so declared figures in instructions for use remain defensible
- Run adversarial test cycles against prompt injection, tool-call manipulation, and retrieval poisoning on a recurring schedule, not a one-time gate
- Treat tool-call permission scopes as a testable artifact, periodically verifying that actual enforcement matches the declared least-privilege policy
- Capture every guardrail intervention (blocked call, fallback triggered, escalation raised) in a retrievable log rather than ephemeral runtime state
- Document the interim methodology used in the absence of finalized CEN-CENELEC harmonized standards, referencing frameworks such as ENISA's AI cybersecurity guidance or NIST's AI RMF
Governance Considerations and Open Gaps
Most high-risk AI system obligations under the AI Act, including Article 15, apply from 2 August 2026 for Annex III systems, with Annex I regulated products given until 2 August 2027 under Article 113. Governance programs should confirm which deadline applies to their specific agent deployment rather than assuming a uniform date. A more immediate gap is standardization: the European Commission issued standardization request M/593 to CEN-CENELEC, and JTC 21 is developing harmonized standards intended to operationalize Article 15 conformity assessment, but these had not been finalized or published as harmonized in the material reviewed for this guide. Article 15's text also does not define a quantitative threshold for "appropriate level" of accuracy, robustness, or cybersecurity, leaving organizations to interpret the standard pending further guidance. In this gap, the defensible position is not to wait, but to document a reasoned interim methodology, apply it consistently, and retain the evidence trail so that when harmonized standards or AI Office guidance arrive, the organization can demonstrate a documented, auditable basis for its prior compliance decisions rather than an undocumented gap.
Interim methodology note
Pending finalized harmonized standards, document the reasoning behind chosen accuracy thresholds, test cadences, and referenced frameworks (such as ENISA guidance or NIST's AI RMF) so the basis for compliance decisions is auditable even before standardization completes.