mlsanityData to 14 Oct 2026

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

arXiv 2603.24755 ↗First posted 25 Mar 2026cs.SE cs.AI cs.CLcited by 30

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 54 of its 86 references that are in the map. First posted on arXiv 25 Mar 2026.

Builds on

54 of 86 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents17 Mar
  2. 2026SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration4 Mar
  3. 2026SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale27 Feb
  4. 2026Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents26 Feb
  5. 2026What to Cut? Predicting Unnecessary Methods in Agentic Code Generation19 Feb
  6. 2026LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces15 Feb
  7. 2026FeatureBench: Benchmarking Agentic Coding for Complex Feature Development11 Feb
  8. 2026SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents10 Feb
  9. 2026Why Agentic-PRs Get Rejected: A Comparative Study of Coding Agents4 Feb
  10. 2026ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development2 Feb
  11. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios20 Dec
  12. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents14 Dec
5 more
  1. 2025Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development6 Nov
  2. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents11 Apr
  3. 2025ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation10 Mar
  4. 2024Evaluating Software Development Agents: Patch Patterns, Code Quality, and Issue Complexity in Real-World GitHub Scenarios16 Oct
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May

Other references · 37

  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 585
  4. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  5. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  6. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
  7. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 58
  8. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 53
  9. 2024Commit0: Library Generation from Scratchcited by 52
  10. 2025LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarkscited by 37
  11. 2023Measuring The Impact Of Programming Language Distributioncited by 32
  12. 2024HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generationcited by 28
25 more
  1. 2025RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedbackcited by 25
  2. 2020Code Smells and Refactoring: A Tertiary Systematic Review of Challenges and Observationscited by 18
  3. 2025CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generationcited by 17
  4. 2025Human-Written vs. AI-Generated Code: A Large-Scale Study of Defects, Vulnerabilities, and Complexitycited by 16
  5. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 15
  6. 2025MaintainCoder: Maintainable Code Generation Under Dynamic Requirementscited by 14
  7. 2017Do Code Clones Matter?cited by 12
  8. 2026SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmarkcited by 12
  9. 2023GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBenchcited by 11
  10. 2025A Taxonomy of Inefficiencies in LLM-Generated Python Codecited by 10
  11. 2026Vibe Code Bench: Evaluating AI Models on End-to-End Web Application Developmentcited by 9
  12. 2025A Hierarchical and Evolvable Benchmark for Fine-Grained Code Instruction Following with Multi-Turn Feedbackcited by 7
  13. 2025SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinementcited by 5
  14. 2026IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Taskscited by 4
  15. 2026Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgementcited by 4
  16. 2026More Rounds, More Noise: Why Multi-Turn Review Fails to Improve Cross-Context Verificationcited by 4
  17. 2026A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactoringscited by 3
  18. 2026From Restructuring to Stabilization: A Large-Scale Experiment on Iterative Code Readability Refactoring with Large Language Modelscited by 3
  19. 2026Novice Developers Produce Larger Review Overhead for Project Maintainers while Vibe Codingcited by 3
  20. 2021Understanding Software Architecture Erosion: A Systematic Mapping Studycited by 2
  21. 2025Investigating Software Aging in LLM-Generated Software Systemscited by 2
  22. 2021Architectural Decay as Predictor of Issue- and Change-Pronenesscited by 1
  23. 2025Show and Tell: Prompt Strategies for Style Control in Multi-Turn LLM Code Generationcited by 1
  24. 2026SCAFFOLD-CEGIS: Preventing Latent Security Degradation in LLM-Driven Iterative Code Refinementcited by 1
  25. 2026An Empirical Study of Interaction Smells in Multi-Turn Human-LLM Collaborative Code Generationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 30.

30 papers.

Sort:

By area:

  1. 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environmentscited by 17
  2. 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgradescited by 7
  3. 2026Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvementcited by 4
  4. 2026LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluationcited by 3
  5. 2026SWE-Touch: Benchmarking Coding Agents When Users Touch the Codecited by 3
  6. 2026Repo0: Design-Driven Zero-to-All Code Generationcited by 3
  7. 2026LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineeringcited by 3
  8. 2026Is Agent Code Less Maintainable Than Human Code?cited by 2
  9. 2026EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environmentscited by 2
  10. 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?cited by 2
  11. 2026Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Studycited by 1
  12. 2026StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turnscited by 1
18 more
  1. 2026Terminal Agents: A Survey of AI Agents in Command-Line Environmentscited by 1
  2. 2026SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproductioncited by 1
  3. 2026EvoGenUI-Bench: Evaluating LLMs as Multi-Turn Generative UI Assistantscited by 1
  4. 2026TRIM: Reducing AI-Generated CodeSlop via Agent Trajectory Minimizationcited by 0
  5. 2026Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performancecited by 0
  6. 2026Persistent Recursive Worlds Enable Autonomous Software Evolutioncited by 0
  7. 2026Super Library Agent: Joint Generation and Maintenance of Multiple Applications Beyond the Single Codebasecited by 0
  8. 2026If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMscited by 0
  9. 2026Test-Driven Approaches to Software Engineering with Large Language Models: A Survey of Phases, Tasks, and Agent Skillscited by 0
  10. 2026Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanismscited by 0
  11. 2026Do Coding Agents Reuse Existing Code or Reinvent the Wheel?cited by 0
  12. 2026StateTape: Action-Conditioned Evidence Lifecycle Modeling for Long-Horizon Coding Agentscited by 0
  13. 2026Can Agents Design Libraries for Agents?cited by 0
  14. 2026SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interactioncited by 0
  15. 2026EvoWeb: A Benchmark for Coding Agents on Evolving Web Applicationscited by 0
  16. 2026StateHarness: Maintaining Evolving Project State for Long-Horizon Coding Agentscited by 0
  17. 2026SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agentscited by 0
  18. 2026Definition Grounding in Theory-Level Auto-formalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
  2. 2026PERFOPT-Bench: Evaluating Coding Agents on Software Performance Optimization
  3. 2026RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
  4. 2026Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering
  5. 2026SWE-Duel: Towards a Self-Scaling Arena for Coding Agents
  6. 2026Beyond Resolution Rates: Behavioral Drivers of Coding Agent Success and Failure
  7. 2026Latent Programming Horizons in Coding Agents
  8. 2026RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents