mlsanityData to 14 Oct 2026

RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Code

arXiv 2503.07832 ↗First posted 10 Mar 2025cs.AI cs.CL cs.LGcited by 30

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent advances in language model (LM) agents and function calling have enabled autonomous, feedback-driven systems to solve problems across various digital domains. To better understand the unique limitations of LM agents, we introduce RefactorBench, a benchmark consisting of 100 large handcrafted multi-file refactoring tasks in popular open-source repositories. Solving tasks within RefactorBench requires thorough exploration of dependencies across multiple files and strong adherence to relevant instructions. Every task is defined by 3 natural language instructions of varying specificity and is mutually exclusive, allowing for the creation of longer combined tasks on the same repository. Baselines on RefactorBench reveal that current LM agents struggle with simple compositional tasks, solving only 22% of tasks with base instructions, in contrast to a human developer with short time constraints solving 87%. Through trajectory analysis, we identify various unique failure modes of LM agents, and further explore the failure mode of tracking past actions. By adapting a baseline agent to condition on representations of state, we achieve a 43.9% improvement in solving RefactorBench tasks. We further extend our state-aware approach to encompass entire digital environments and outline potential directions for future research. RefactorBench aims to support the study of LM agents by providing a set of real-world, multi-hop tasks within the realm of code.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 58 of its 79 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

58 of 79 references.

References · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
38 more
  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  3. 2022A Generalist Agentcited by 808
  4. 2024A Survey on Large Language Models for Code Generationcited by 658
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  6. 2023Cognitive Architectures for Language Agentscited by 429
  7. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  8. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  9. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 366
  10. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  11. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  12. 2024AutoCodeRover: Autonomous Program Improvementcited by 278
  13. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  14. 2024AI Agents That Mattercited by 219
  15. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  16. 2024OmniParser for Pure Vision Based GUI Agentcited by 198
  17. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 171
  18. 2024A Human-Inspired Reading Agent with Gist Memory of Very Long Contextscited by 146
  19. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  20. 2021The Benchmark Lotterycited by 136
  21. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
  22. 2024Language Agents as Optimizable Graphscited by 89
  23. 2024Planning In Natural Language Improves LLM Search For Code Generationcited by 83
  24. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 65
  25. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 56
  26. 2024MASAI: Modular Architecture for Software-engineering AI Agentscited by 49
  27. 2023Entity Tracking in Language Modelscited by 44
  28. 2024Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Studycited by 44
  29. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Modelscited by 37
  30. 2024Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agentscited by 36
  31. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  32. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  33. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 25
  34. 2024AutoDev: Automated AI-Driven Developmentcited by 24
  35. 2022Insights From the NeurIPS 2021 NetHack Challengecited by 20
  36. 2021Neural Recursive Belief States in Multi-Agent Reinforcement Learningcited by 18
  37. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  38. 2024Selective Prompt Anchoring for Code Generationcited by 16

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 19.

30 papers.

Sort:

By area:

  1. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  2. 2025The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reasoncited by 49
  3. 2025When Agents go Astray: Course-Correcting SWE Agents with PRMscited by 28
  4. 2025RPG: A Repository Planning Graph for Unified and Scalable Codebase Generationcited by 19
  5. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  6. 2025ARC-GEN: A Mimetic Procedural Benchmark Generator for the Abstraction and Reasoning Corpuscited by 10
  7. 2026CodeTaste: Can LLMs Generate Human-Level Code Refactorings?cited by 8
  8. 2026Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptationcited by 7
  9. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolutioncited by 6
  10. 2026SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoringcited by 6
  11. 2025Refactoring Codebases through Library Designcited by 4
  12. 2025Gistify! Codebase-Level Understanding via Runtime Executioncited by 4
18 more
  1. 2025PerfBench: Can Agents Resolve Real-World Performance Bugs?cited by 3
  2. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3
  3. 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluationcited by 3
  4. 2025PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguisticscited by 2
  5. 2026PseudoAct: Leveraging Pseudocode Synthesis for Flexible Planning and Action Control in Large Language Model Agentscited by 2
  6. 2026Is Agent Code Less Maintainable Than Human Code?cited by 2
  7. 2026Lean Refactor: Multi-Objective Controllable Proof Optimization via Agentic Strategy Searchcited by 1
  8. 2026Multi-Agent Coordinated Rename Refactoringcited by 0
  9. 2026Theory of Code Space: Do Code Agents Understand Software Architecture?cited by 0
  10. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0
  11. 2026SmellBench: Towards Fine-Grained Evaluation of Code Agents on Refactoring Taskscited by 0
  12. 2026What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Revealcited by 0
  13. 2026CROCODIL: Cross-Model Code Editing with LLMscited by 0
  14. 2026RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agentscited by 0
  15. 2026AsynCodeBench: Benchmarking Collaboration of Asynchronous Multi-Agent Systems in Software Engineeringcited by 0
  16. 2026Can Agents Design Libraries for Agents?cited by 0
  17. 2026StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agentscited by 0
  18. 2026LintEval: Benchmarking LLMs on Linter-Guided File Refactoringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Grow the Harness, Not the Context: From Strategy-Free Scaffolds to Reusable Specialist Agents
  2. 2026AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
  3. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  4. 2025On the Limits of Innate Planning in Large Language Models
  5. 2025A Self-Improving Coding Agent
  6. 2024MapCoder: Multi-Agent Code Generation for Competitive Problem Solving
  7. 2025PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases
  8. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios