Iterative Audit Convergence in LLM-Managed Multi-Agent Systems: A Case Study in Prompt-Engineering Quality Assurance
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Prompt specifications for multi-agent large language model (LLM) systems carry data contracts and integration logic across interdependent files but are rarely subjected to structured-inspection rigor. We report a single-system case study of iterative, agent-driven auditing applied to AEGIS (Autonomous Engineering Governance and Intelligence System), a seven-lane production pipeline whose 7152-line specification surface was audited across nine rounds, surfacing 51 consistency defects (per-round counts of 15, 8, 12, 2, 8, 1, 4, 1, 0). We present a seven-category post hoc taxonomy with explicit coding rules, non-monotonic convergence consistent with cascading edits and audit-scope expansion, and a locked audit protocol. We further report two partial replications on a public synthetic mini-specification: a cross-LLM panel of four frontier vendors (OpenAI, Anthropic, Google, xAI; 12 traces; multi-vendor union detects all five seeded defects) and an inter-rater reliability check on a stratified subsample (Cohen's kappa = 0.80 on category, 0.46 on severity). The full reproducibility bundle accompanies the submission.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 28 references that are in the map. First posted on arXiv 12 May 2026.
Builds on
14 of 28 references.
Closest ancestors · 5
Same direction, newest first
- 2026From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level7 Jan
- 2026The Rise of Agentic Testing: Multi-Agent Systems for Robust Software Quality Assurance5 Jan
- 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
- 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead7 Apr
Other references · 9
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
- 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
- 2024Leveraging LLMs for the Quality Assurance of Software Requirementscited by 15
- 2025Beyond Task Completion: An Assessment Framework for Evaluating Agentic AI Systemscited by 7
- 2025Promptware Engineering: Software Engineering for Prompt-Enabled Systemscited by 6
- 2025A Blueprint for AI-Driven Software Quality: Integrating LLMs with Established Standardscited by 1
- 2026Closed-Loop Autonomous Software Development via Jira-Integrated Backlog Orchestration: A Case Study in Deterministic Control and Safety-Constrained Automationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Recursive Knowledge Synthesis for Multi-LLM Systems: Stability Analysis and Tri-Agent Audit Framework
- 2026A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents
- 2026AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems
- 2026From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
- 2026Auditing Action Settlement in LLM Agent Environments: Order, Progress, and Replay
- 2026Maat: Independent Deterministic Contract-Based Governance for Multi-Agent LLM Workflows
- 2025Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
- 2026BRA-Audit: Budgeted Runtime Auditing for LLM Multi-Agent Systems via Cumulative-Exposure Audit-Point Placement