SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
On the map: LLM agents and tool use → Software engineering coding agents
Abstract (arXiv)
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 86 references that are in the map. First posted on arXiv 25 Mar 2026.
Builds on
54 of 86 references.
Closest ancestors · 17
Same direction, newest first
- 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents17 Mar
- 2026SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration4 Mar
- 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale27 Feb
- 2026Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents26 Feb
- 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generation19 Feb
- 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces15 Feb
- 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Development11 Feb
- 2026SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents10 Feb
- 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents4 Feb
- 2026ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development2 Feb
- 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios20 Dec
- 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents14 Dec
5 more
- 2025Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development6 Nov
- 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents11 Apr
- 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation10 Mar
- 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios16 Oct
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
Other references · 37
- 2021Program Synthesis with Large Language Modelscited by 3,637
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
- 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
- 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
- 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 58
- 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
- 2024Commit0: Library Generation from Scratchcited by 52
- 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
- 2023Measuring The Impact Of Programming Language Distributioncited by 32
- 2024HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generationcited by 28
25 more
- 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 25
- 2020Code Smells and Refactoring: A Tertiary Systematic Review of Challenges and Observationscited by 18
- 2025CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generationcited by 17
- 2025Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexitycited by 16
- 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 15
- 2025MaintainCoder: Maintainable Code Generation Under Dynamic Requirementscited by 14
- 2017Do Code Clones Matter?cited by 12
- 2026SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmarkcited by 12
- 2023GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBenchcited by 11
- 2025A Taxonomy of Inefficiencies in LLM-Generated Python Codecited by 10
- 2026Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Developmentcited by 9
- 2025A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedbackcited by 7
- 2025SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinementcited by 5
- 2026IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Taskscited by 4
- 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
- 2026More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verificationcited by 4
- 2026A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactoringscited by 3
- 2026From Restructuring to Stabilization: A Large-Scale Experiment on Iterative Code Readability Refactoring with Large Language Modelscited by 3
- 2026Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Codingcited by 3
- 2021Understanding Software Architecture Erosion: A Systematic Mapping Studycited by 2
- 2025Investigating Software Aging in LLM-Generated Software Systemscited by 2
- 2021Architectural Decay as Predictor of Issue- and Change-Pronenesscited by 1
- 2025Show and Tell: Prompt Strategies for Style Control in Multi-Turn LLM Code Generationcited by 1
- 2026SCAFFOLD-CEGIS: Preventing Latent Security Degradation in LLM-Driven Iterative Code Refinementcited by 1
- 2026An Empirical Study of Interaction Smells in Multi-Turn Human-LLM Collaborative Code Generationcited by 1
Built on by
Citing papers per year: 2026: 30.
30 papers.
Sort:
By area:
- 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environmentscited by 17
- 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgradescited by 7
- 2026Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvementcited by 4
- 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluationcited by 3
- 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 3
- 2026Repo0: Design-Driven Zero-to-All Code Generationcited by 3
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 3
- 2026Is Agent Code Less Maintainable Than Human Code?cited by 2
- 2026EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environmentscited by 2
- 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?cited by 2
- 2026Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Studycited by 1
- 2026StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turnscited by 1
18 more
- 2026Terminal Agents: A Survey of AI Agents in Command-Line Environmentscited by 1
- 2026SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproductioncited by 1
- 2026EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistantscited by 1
- 2026TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimizationcited by 0
- 2026Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performancecited by 0
- 2026Persistent Recursive Worlds Enable Autonomous Software Evolutioncited by 0
- 2026Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebasecited by 0
- 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMscited by 0
- 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skillscited by 0
- 2026Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanismscited by 0
- 2026Do Coding Agents Reuse Existing Code or Reinvent the Wheel?cited by 0
- 2026StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agentscited by 0
- 2026Can Agents Design Libraries for Agents?cited by 0
- 2026SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interactioncited by 0
- 2026EvoWeb: A Benchmark for Coding Agents on Evolving Web Applicationscited by 0
- 2026StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agentscited by 0
- 2026SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agentscited by 0
- 2026Definition Grounding in Theory-Level Auto-formalizationcited by 0
- 2026Definition Grounding in Theory-Level Auto-formalization13 Oct 2026
- 2026SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents13 Oct 2026
- 2026StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents13 Oct 2026
- 2026EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications12 Oct 2026
- 2026SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction8 Oct 2026
- 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?6 Oct 2026
- 2026Can Agents Design Libraries for Agents?29 Sep 2026
- 2026StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agents28 Sep 2026
- 2026Do Coding Agents Reuse Existing Code or Reinvent the Wheel?28 Sep 2026
- 2026Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms23 Sep 2026
- 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills10 Sep 2026
- 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs9 Sep 2026
18 more
- 2026Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement1 Sep 2026
- 2026EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants29 Aug 2026
- 2026Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebase29 Aug 2026
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering28 Aug 2026
- 2026SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction25 Aug 2026
- 2026Terminal Agents: A Survey of AI Agents in Command-Line Environments20 Aug 2026
- 2026Repo0: Design-Driven Zero-to-All Code Generation20 Aug 2026
- 2026Persistent Recursive Worlds Enable Autonomous Software Evolution11 Aug 2026
- 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Code3 Aug 2026
- 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation31 Jul 2026
- 2026Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance29 Jul 2026
- 2026TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimization20 Jul 2026
- 2026EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments2 Jul 2026
- 2026Is Agent Code Less Maintainable Than Human Code?19 Jun 2026
- 2026StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns17 Jun 2026
- 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments4 Jun 2026
- 2026Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study19 May 2026
- 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades14 May 2026
- 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgradescited by 7
- 2026Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvementcited by 4
- 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluationcited by 3
- 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 3
- 2026Repo0: Design-Driven Zero-to-All Code Generationcited by 3
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 3
- 2026Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Studycited by 1
- 2026StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turnscited by 1
- 2026Terminal Agents: A Survey of AI Agents in Command-Line Environmentscited by 1
- 2026SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproductioncited by 1
- 2026TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimizationcited by 0
- 2026Persistent Recursive Worlds Enable Autonomous Software Evolutioncited by 0
12 most cited of 21 All 21 (JSON)
- 2026SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents13 Oct 2026
- 2026StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agents13 Oct 2026
- 2026EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications12 Oct 2026
- 2026SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction8 Oct 2026
- 2026StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agents28 Sep 2026
- 2026Do Coding Agents Reuse Existing Code or Reinvent the Wheel?28 Sep 2026
- 2026Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms23 Sep 2026
- 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skills10 Sep 2026
- 2026Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement1 Sep 2026
- 2026Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebase29 Aug 2026
- 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering28 Aug 2026
- 2026SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction25 Aug 2026
12 newest of 21 All 21 (JSON)
- 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environmentscited by 17
- 2026Is Agent Code Less Maintainable Than Human Code?cited by 2
- 2026EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environmentscited by 2
- 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?cited by 2
- 2026EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistantscited by 1
- 2026Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performancecited by 0
- 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMscited by 0
- 2026Can Agents Design Libraries for Agents?cited by 0
- 2026Definition Grounding in Theory-Level Auto-formalizationcited by 0
- 2026Definition Grounding in Theory-Level Auto-formalization13 Oct 2026
- 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?6 Oct 2026
- 2026Can Agents Design Libraries for Agents?29 Sep 2026
- 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs9 Sep 2026
- 2026EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistants29 Aug 2026
- 2026Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance29 Jul 2026
- 2026EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments2 Jul 2026
- 2026Is Agent Code Less Maintainable Than Human Code?19 Jun 2026
- 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments4 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
- 2026PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization
- 2026RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
- 2026Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
- 2026SWE-Duel: Towards a Self-Scaling Arena for Coding Agents
- 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
- 2026Latent Programming Horizons in Coding Agents
- 2026RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents