mlsanityData to 7 Oct 2026

WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

arXiv 2609.36635 ↗First posted 29 Sep 2026cs.SE cs.AIcited by 0

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness and exposes faulty behavior during execution. Such evidence makes audit findings actionable, yet benchmark evaluation is difficult when cases reuse public historical bugs and witnesses or require manual construction. We present WitnessGym, an automated framework for constructing bug-validation benchmarks through bug injection. It injects bugs into test-reached paths of real projects, rebuilds each project, and retains cases exposed by a construction-time witness. Bug specifications and execution adapters allow extension to additional bug types and languages. Bug-preserving transformations vary the surrounding structure while preserving the witness behavior. Based on real-world Java projects with test suites, WitnessGym automatically constructs 1,300 benchmark cases. The injected patches resemble historical bug patches and are difficult for the two evaluated models to distinguish in blinded comparisons. We evaluate four coding agent frameworks in six framework/model pairings across bug types, execution contexts, and transformation depths. Witness construction remains difficult even when the bug pattern is known. Our framework, benchmark cases, and evaluation scripts are available.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 54 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

31 of 54 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments30 Jul
  2. 2026AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection13 Apr
  3. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?17 Nov
  4. 2025Agentic Bug Reproduction for Effective Automated Program Repair at Google3 Feb
  5. 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing30 Jan
  6. 2024SWE-Search: Enhancing Software Agents with Monte Carlo Tree Search and Iterative Refinement26 Oct
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agents23 Jul
  8. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 23

  1. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  2. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  3. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  4. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  5. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  6. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  7. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
  8. 2025BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemscited by 37
  9. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  10. 2023Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Promptingcited by 23
  11. 2024LLMDFA: Analyzing Dataflow in Code with Large Language Modelscited by 22
  12. 2025Issue2Test: Generating Reproducing Test Cases from Issue Reportscited by 18
11 more
  1. 2024LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programscited by 14
  2. 2025PoCGen: Generating Proof-of-Concept Exploits for Vulnerabilities in Npm Packagescited by 12
  3. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 8
  4. 2025PoCo: Agentic Proof-of-Concept Exploit Generation for Smart Contractscited by 7
  5. 2025Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysiscited by 4
  6. 2020EvilCoder: Automated Bug Insertioncited by 2
  7. 2025Dynamic Evaluation for Oversensitivity in LLMscited by 2
  8. 2025Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Modelscited by 1
  9. 2026PoC-Gym: Towards More Reliable LLM-Assisted Proof-of-Concept Exploit Generationcited by 1
  10. 2026SecCodeBench-V2 Technical Reportcited by 1
  11. 2026V2E: Validating Smart Contract Vulnerabilities through Profit-driven Exploit Generation and Executioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution
  2. 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?
  3. 2026Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports
  4. 2026Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution
  5. 2026GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
  6. 2026TestExplora: Benchmarking LLMs for Proactive Bug Discovery via Repository-Level Test Generation
  7. 2026SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?
  8. 2026AgentSZZ: Teaching the LLM Agent to Play Detective with Bug-Inducing Commits