mlsanityData to 7 Oct 2026

Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

arXiv 2610.01023 ↗First posted 1 Oct 2026cs.SE cs.AI cs.CLcited by 1

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Coding agents can return plausible patches that omit required behavior. These failures are hard to review because long traces and confident summaries often hide what was missed. We ask when a nominally weaker reviewer can reliably decide whether a patch solves its issue. We study 411 execution-labeled traces from three agents and 101 controlled cases. On 154 GPT-5.4 traces, structured but unchecked evidence raises both defect catch and over-rejection. We then provide official execution evidence as an upper-bound diagnostic. After choosing and freezing one of two formats per reviewer, five of six reviewers improve both rates on 122 held-out traces; two classify every trace correctly. Reviewer size is not a consistent predictor of quality. Because official tests are unavailable in deployment, we also evaluate a frozen cascade with patch-caused static errors and generated tests that first fail on the unpatched repository. On 121 scored held-out GPT-5.4 traces and 59 Gemini traces, its coverage is 0.89 and 0.86, risk is 0.33 and 0.26, catch is 0.76 and 0.80, and over-rejection is 0.66 and 0.67. Most false rejections occur when unresolved cases reach the reviewer. Official execution evidence shows the potential of weak review when decisive checks are available. Producing equally reliable checks without official tests remains the main bottleneck.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 39 of its 64 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

39 of 64 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026The End of Code Review: Coding Agents Supersede Human Inspection11 Jun
  2. 2026SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents20 May
  3. 2026Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents8 Feb
  4. 2026Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair27 Jan
  5. 2025UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench10 Jun
  6. 2025TRAIL: Trace Reasoning and Agentic Issue Localization13 May
  7. 2025Large Language Model Critics for Execution-Free Evaluation of Code Changes28 Jan
  8. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  9. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents18 Jun

Other references · 30

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  3. 2017Selective Classification for Deep Neural Networkscited by 799
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  6. 2018AI safety via debatecited by 384
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  10. 2020Selective Question Answering under Domain Shiftcited by 228
  11. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  12. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
18 more
  1. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  2. 2024LLM Critics Help Catch LLM Bugscited by 138
  3. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  4. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  5. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  6. 2024Prover-Verifier Games improve legibility of LLM outputscited by 79
  7. 2024Don't Trust: Verify -- Grounding LLM Quantitative Reasoning with Autoformalizationcited by 53
  8. 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
  9. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoringcited by 49
  10. 2025Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Studycited by 47
  11. 2025Ctrl-Z: Controlling AI Agents via Resamplingcited by 40
  12. 2025Adaptive Attacks on Trusted Monitors Subvert AI Control Protocolscited by 27
  13. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  14. 2025Mutation-Guided LLM-based Test Generation at Metacited by 23
  15. 2026LLM-as-a-Verifier: A General-Purpose Verification Frameworkcited by 21
  16. 2017Identifying Patch Correctness in Test-Based Program Repaircited by 16
  17. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 10
  18. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineeringcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
  2. 2026All Smoke, No Alarm: Oracle Signals in Agent-Authored Test Code
  3. 2026Coding Agents Don't Know When to Act
  4. 2026TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
  5. 2026SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
  6. 2026Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch
  7. 2026Building to the Test: Coding Agents Deliver What You Check, Not What You Requested
  8. 2026Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair