Cross-Provider Review as a Runtime Contract for Coding Agents: A Controlled Pilot and Fault-Injection Study
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Coding agents increasingly share a workstation while drawing on separate providers and subscription allowances. A second agent can inspect a completed answer, but the call spends another pool and may provide no substantive finding. We describe an advisory cross-provider review contract: distinct resource pools, bounded execution, restricted reviewer capabilities, complete input delivery, usable semantic output, explicit failure states and durable per-attempt evidence. In a controlled, agent-authored pilot of 20 paired development turns, eight had a material reviewer finding (95% exact interval 19.1-63.9%). A boundary-condition scan across both reviewer backends reproduced a previously discovered false success on partial input: four truncation levels passed historically and failed after repair. The scan also found and repaired cancellation during process reaping. In real CLI probes, Claude had no writing tools; Codex attempted writes in five of five read-only trials, each write tool failed, and no disposable repository changed. These tests cover specified paths and versions, not field reliability. A preregistered shadow study of metadata-only review allocation accrued 25 formal observations before an exact-runtime regression found a third defect: a reviewer exiting nonzero with a well-formed verdict was counted as complete. Exit status was not recorded per attempt, so exposure cannot be resolved retrospectively. The 25 formal and two pending records remain an audit cohort; the measurement-valid cohort restarted at zero and collection has begun. No gate result is reported.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 23 references that are in the map. First posted on arXiv 7 Oct 2026.
Builds on
21 of 23 references.
Closest ancestors · 6
Same direction, newest first
- 2026Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents1 Oct
- 2026From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents9 Jul
- 2026AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents13 May
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
- 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 15
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
- 2023AutoMix: Automatically Mixing Language Modelscited by 116
- 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
- 2025Rethinking Code Review Workflows with LLM Assistance: An Empirical Studycited by 10
- 2026Cross-Model LLM Code Review: Should you use Claude to review Codex or vice versa?cited by 2
- 2026When to Screen, When to Bypass: LLM-Judges in Resource-Scarce AI-Human Workflowcited by 1
- 2026Think Again or Think Longer? Selective Verification for Budget-Aware Reasoningcited by 1
3 more
- 2026Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agentscited by 1
- 2026Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injectioncited by 1
- 2026When Does a Second Model Help? Cross-Model Review in LLM Verificationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026From Industry Claims to Empirical Reality: An Empirical Study of Code Review Agents in Pull Requests
- 2026SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
- 2026SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review
- 2026The End of Code Review: Coding Agents Supersede Human Inspection
- 2026Software Delegation Contracts: Measuring Reviewability in AI Coding-Agent Work
- 2026When AI Teammates Meet Code Review: Collaboration Signals Shaping the Integration of Agent-Authored Pull Requests
- 2026Probe-and-Refine Tuning of Repository Guidance for Coding Agents
- 2026Agentic Code Review in the Terminal: A Trajectory-Level Analysis of Behavior, Cost, and Human-Alignment