mlsanityData to 14 Oct 2026

UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench

arXiv 2506.09289 ↗First posted 10 Jun 2025cs.SE cs.CLcited by 45

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

The advent of Large Language Models (LLMs) has spurred the development of coding agents for real-world code generation. As a widely used benchmark for evaluating the code generation capabilities of these agents, SWE-Bench uses real-world problems based on GitHub issues and their corresponding pull requests. However, the manually written test cases included in these pull requests are often insufficient, allowing generated patches to pass the tests without resolving the underlying issue. To address this challenge, we introduce UTGenerator, an LLM-driven test case generator that automatically analyzes codebases and dependencies to generate test cases for real-world Python projects. Building on UTGenerator, we propose UTBoost, a comprehensive framework for test case augmentation. In our evaluation, we identified 36 task instances with insufficient test cases and uncovered 345 erroneous patches incorrectly labeled as passed in the original SWE Bench. These corrections, impacting 40.9% of SWE-Bench Lite and 24.4% of SWE-Bench Verified leaderboard entries, yield 18 and 11 ranking changes, respectively.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 10 of its 41 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

10 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios16 Oct
  2. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scale9 Sep

Other references · 8

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  5. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
  6. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 91
  7. 2024Infant Agent: A Tool-Integrated, Logic-Driven Agent with Cost-Effective API Usagecited by 11
  8. 2022Intramorphic Testing: A New Approach to the Test Oracle Problemcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 40.

45 papers.

Sort:

By area:

  1. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  2. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfacescited by 34
  3. 2026BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarkscited by 15
  4. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  5. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 14
  6. 2026Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboardscited by 13
  7. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 12
  8. 2026FALAT: Tracing Failures in LLM Agent Trajectories via Dependency-Guided Searchcited by 12
  9. 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agentscited by 8
  10. 2025SWE-Factory: Your Automated Factory for Issue Resolution Training Data and Evaluation Benchmarkscited by 7
  11. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Codecited by 7
  12. 2025Agentic Software Issue Resolution with Large Language Models: A Surveycited by 6
33 more
  1. 2026ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Developmentcited by 6
  2. 2026Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repaircited by 6
  3. 2025Scalable Supervision for Software Agents via Patch Reasoningcited by 5
  4. 2026Noisy Data is Destructive to Reinforcement Learning with Verifiable Rewardscited by 5
  5. 2026SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cyclecited by 5
  6. 2026Anchor: Mitigating Artifact Drift in Agent Benchmark Generationcited by 5
  7. 2026What Resolve Rate Hides: Trajectory Structure Diagnostics for Coding Agentscited by 5
  8. 2026CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generationcited by 4
  9. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agentscited by 3
  10. 2026SciCode-Verified: How Benchmark Defects Underestimated the Scientific-Coding Ability of Language Modelscited by 3
  11. 2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 3
  12. 2026ExecCritic: Learn to Test, Test to Improve for Coding Agentscited by 3
  13. 2026APEX-SWEcited by 2
  14. 2026SkillForge: Self-Distilling Agents for Project-Specific Issue Resolutioncited by 2
  15. 2026Do Agent Benchmarks Do What They Say? An Executable-Contract Audit of Tool-Using Agent Environmentscited by 2
  16. 2026Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguitycited by 1
  17. 2026RESCORE: LLM-Driven Simulation Recovery in Control Systems Research Paperscited by 1
  18. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  19. 2026How Much Static Structure Do Code Agents Need? A Study of Deterministic Anchoringcited by 1
  20. 2026SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvementscited by 1
  21. 2026Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarkscited by 1
  22. 2026PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarkscited by 1
  23. 2026MEGA: Self-Evolving Agent Optimization Infrastructure via Wisdom Graphcited by 1
  24. 2026QuoteBench: How Matched Scores Can Hide Command-Path Failurescited by 1
  25. 2026SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agentscited by 1
  26. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  27. 2026Before the Rollout Ends: Early Terminal Reward Prediction for Long-horizon Coding Agentscited by 1
  28. 2026Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agentscited by 1
  29. 2026No Accidental Software Agent First Canonical Code for Human Code Entropy Reduction and 30 to 500 times Lower Frontier Model Requirementscited by 0
  30. 2026What Makes Software Issue Resolution Tasks Difficult for Agents?cited by 0
  31. 2026Schr"odinger's Code Repository: Have LLMs Learned SWE-bench or Memorized It?cited by 0
  32. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  33. 2026Re-executing Coding-Agent Evaluations: Unchanged Decisions and Unresolved Policy Riskcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models as Test Case Generators: Performance Evaluation and Enhancement
  2. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents
  3. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
  4. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
  5. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
  6. 2025Learning to Generate Unit Tests for Automated Debugging
  7. 2026TDD-Agent: Test-Driven Reasoning for Code Generation
  8. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring