# A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI

arXiv: https://arxiv.org/abs/2608.04921 · HTML: https://mlsanity.com/paper/2608.04921/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 65 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 12 of 12)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [AI auditing: The Broken Bus on the Road to AI Accountability](https://mlsanity.com/paper/2401.14462/) (2024)
- [Hype, Sustainability, and the Price of the Bigger-is-Better Paradigm in AI](https://mlsanity.com/paper/2409.14160/) (2024)
- [Model-as-a-Service (MaaS): A Survey](https://mlsanity.com/paper/2311.05804/) (2023)
- [Algorithmic Auditing and Social Justice: Lessons from the History of Audit Studies](https://mlsanity.com/paper/2109.06974/) (2021)
- [A Silicon Valley Love Triangle: Hiring Algorithms, Pseudo-Science, and the Quest for Auditability](https://mlsanity.com/paper/2106.12403/) (2021)
- [System Cards for AI-Based Decision-Making for Public Policy](https://mlsanity.com/paper/2203.04754/) (2022)
- [Massively Digitized Power Grid: Opportunities and Challenges of Use-inspired AI](https://mlsanity.com/paper/2205.05180/) (2022)
- [Measuring What Matters: Connecting AI Ethics Evaluations to System Attributes, Hazards, and Harms](https://mlsanity.com/paper/2510.10339/) (2025)
- [A Blueprint for Auditing Generative AI](https://mlsanity.com/paper/2407.05338/) (2024)
- [A Game-Theoretic Analysis of Auditing Differentially Private Algorithms with Epistemically Disparate Herd](https://mlsanity.com/paper/2404.16195/) (2024)
- [Pragmatic auditing: a pilot-driven approach for auditing Machine Learning systems](https://mlsanity.com/paper/2405.13191/) (2024)

## Abstract (arXiv)

As AI systems become increasingly integrated into diverse interfaces and applications, model-centric audits are insufficient to address risks arising from interactions among system components and deployment environments. System integration has long been central to software audits in safety-critical domains such as aerospace. However, its role in AI auditing remains underexplored. Scanning through 4,259 documents, we present a scoping review of AI audits that treat system integration as a core tenet of evaluation (n = 58). Using reflexive thematic analysis, we analyze their elements, actors, enablers, and constraints. We find that the corpus represents an emerging yet still fragmented form of AI auditing: few existing measures target integration-specific risks; large gaps remain in meeting traditional audit expectations; and access to necessary information and resources significantly influences audit design. Nonetheless, integration can be categorized across three sites (inter-component, system-environment, and multi-system), each serving the functions of risk exploration, risk determination, coordination, and procedural regularity. Deviating from other types of evaluations, these audits assess qualities specific to system integration, including compatibility, completeness, and oversight. This review calls on the AI community to prioritize system integration as a core strategy for addressing AI risk, and to develop audit practices capable of capturing failures across components, environments, and systems beyond the reach of component-level evaluation.