mlsanityData to 7 Oct 2026

AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems

arXiv 2609.37864 ↗First posted 29 Sep 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Agent harness bugs exhibit unique characteristics and remain challenging for state-of-the-art software agents to repair. Progress in this area is further hindered by existing benchmarks, which contain only a small and fixed number of executable harness bugs while requiring hundreds of human hours to construct. This work presents AgentBug-Smith, an automated harness bug reproduction approach that continuously discovers and reproduces real-world harness bugs from open-source agentic systems. Across different backbone LLMs, AgentBug-Smith consistently outperforms existing bug reproduction techniques designed for general software, achieving 10.67% - 27.56% higher success rates of reproducing harness bugs. By applying AgentBug-Smith to open-source agentic systems in the wild, we construct Live-Harness-Bench, a live and extensible benchmark that currently contains 200 reproducible harness bugs. We further demonstrate the utility of Live-Harness-Bench through two downstream applications. First, we use Live-Harness-Bench as the evaluation benchmark to systematically evaluate state-of-the-art software agents, revealing their limited capabilities in repairing real-world harness bugs. Second, we use Live-Harness-Bench as a knowledge base of real-world harness bug fixes, from which reusable repair skills can be distilled to improve existing software agents, increasing their harness-bug repair rates by 6.32%. Together, AgentBug-Smith and Live-Harness-Bench establish a scalable foundation for continuously evaluating and improving software agents on harness bug repair, turning real-world agent failures into executable evaluation instances and reusable knowledge for harness improvement, thus contributing to the ultimate goal of recursively self-improving agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 38 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

30 of 38 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reports17 Jul
  2. 2026Code as Agent Harness18 May
  3. 2026Understanding Bugs in Modern Agentic Frameworks: A Study of Symptoms, Root Causes, and Triggering Conditions10 Apr
  4. 2025Where LLM Agents Fail and How They can Learn From Failures29 Sep
  5. 2025An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications23 Sep
  6. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories23 Jun
  7. 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents26 May
  8. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr
  9. 2025SWE-smith: Scaling Data for Software Engineering Agents30 Apr
  10. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents9 Apr
  11. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  12. 2024Training Software Engineering Agents and Verifiers with SWE-Gym30 Dec
3 more
  1. 2024Large Language Model-Based Agents for Software Engineering: A Survey4 Sep
  2. 2024Agentless: Demystifying LLM-based Software Engineering Agents1 Jul
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 15

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  3. 2025Accelerating scientific discovery with Co-Scientistcited by 353
  4. 2024Agent Workflow Memorycited by 342
  5. 2024AutoCodeRover: Autonomous Program Improvementcited by 272
  6. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  7. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  8. 2023Large Language Models for Robotics: A Surveycited by 132
  9. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  10. 2025SWE-bench Goes Live!cited by 74
  11. 2025Repo2Run: Automated Building Executable Environment for Code Repository at Scalecited by 39
  12. 2024Are LLMs Correctly Integrated into Software Systems?cited by 13
3 more
  1. 2019BugSwarm: Mining and Continuously Growing a Dataset of Reproducible Failures and Fixescited by 12
  2. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  3. 2025FairMedQA: Benchmarking Bias in Large Language Models for Medical Question Answeringcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
  2. 2026ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures
  3. 2026From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws
  4. 2026Complex Agents, Shallow Tests: Demystifying and Enhancing Test Adequacy of Agent Harness in the Wild
  5. 2026AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  6. 2026ChainSWE: Benchmarking Coding Agents on Multi-Bug Software Maintenance
  7. 2024An Empirical Study on LLM-based Agents for Automated Bug Fixing
  8. 2025PerfBench: Can Agents Resolve Real-World Performance Bugs?