# Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Risk

arXiv: https://arxiv.org/abs/2610.14460 · HTML: https://mlsanity.com/paper/2610.14460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 25 of 25)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?](https://mlsanity.com/paper/2509.16941/) (2025)
- [Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving](https://mlsanity.com/paper/2504.02605/) (2025)
- [SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents](https://mlsanity.com/paper/2505.20411/) (2025)
- [SWE-Bench+: Enhanced Coding Benchmark for LLMs](https://mlsanity.com/paper/2410.06992/) (2024)
- [Huxley-G"odel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine](https://mlsanity.com/paper/2510.21614/) (2025)
- [Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study](https://mlsanity.com/paper/2503.15223/) (2025)
- [UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench](https://mlsanity.com/paper/2506.09289/) (2025)
- [SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale](https://mlsanity.com/paper/2602.23866/) (2026)
- [GitBug-Java: A Reproducible Benchmark of Recent Java Bugs](https://mlsanity.com/paper/2402.02961/) (2024)
- [SWE-MiniSandbox: Container-Free Reinforcement Learning for Building Software Engineering Agents](https://mlsanity.com/paper/2602.11210/) (2026)
- [Auditing Reward Hackability in Code RL Training Environments](https://mlsanity.com/paper/2606.16062/) (2026)
- [The Implementation Lottery: Auditing Idea Reliability in Automated Research](https://mlsanity.com/paper/2607.26587/) (2026)
- [SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents](https://mlsanity.com/paper/2609.08149/) (2026)
- [KAT-Coder-V2.5 Technical Report](https://mlsanity.com/paper/2607.05471/) (2026)
- [LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents](https://mlsanity.com/paper/2608.17393/) (2026)
- [GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions](https://mlsanity.com/paper/2310.15642/) (2023)
- [Probe to Generate: Program Variant-Guided Test Augmentation for Repository-Level Repair Benchmarks](https://mlsanity.com/paper/2604.01518/) (2026)
- [Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model](https://mlsanity.com/paper/2608.13867/) (2026)
- [Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR](https://mlsanity.com/paper/2609.01354/) (2026)
- [DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale](https://mlsanity.com/paper/2609.22978/) (2026)
- [Understanding and Detecting Flaky Builds in GitHub Actions](https://mlsanity.com/paper/2602.02307/) (2026)
- [Flaky Test Sanitisation via On-the-Fly Assumption Inference for Tests with Network Dependencies](https://mlsanity.com/paper/2208.01106/) (2022)
- [The Effects of Computational Resources on Flaky Tests](https://mlsanity.com/paper/2310.12132/) (2023)

## Abstract (arXiv)

Executable tests grade large language model (LLM) coding-agent patches for leaderboards, rewards and self-improvement. Two preregistered re-execution studies returned negative results. SWE-bench Pro v1's offline option rejected none of 59 accepted Go patches whose grading configuration changed. Their exploratory, unweighted upper confidence limit is 4.95% under candidate independence, leaving a 3% exposed-patch risk unresolved; the preregistered estimator uses inverse sampling-probability weights. An extension was inconclusive. In a selected stratum, a reconstructed module cache built from the reference solution's dependencies rejected accepted alternatives from four models that passed with their own dependencies. Re-executing 1,929 selected agent-version/task evaluations from the Darwin G"odel Machine's archive preserved best-agent selection and system-variant ranking, with search history and scores of evaluations not rerun fixed. Exploratory host, image and adapted-grader comparisons also preserved them. Seven of 23 disagreements with archived verdicts did not recur on a second host; rebuilt images changed 16 three-run majority verdicts (0.83%) under four-CPU execution. We provide an evidence-to-claim summary, lessons for grader audits and an anonymous replication package. These results concern accepted patches and fixed-history decisions; general policy safety and adaptive-learning robustness require broader evaluations.