mlsanityData to 7 Oct 2026

Agentless: Demystifying LLM-based Software Engineering Agents

arXiv 2407.01489 ↗First posted 1 Jul 2024cs.SE cs.AI cs.CLcited by 400

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost (0.70) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.

Key facts. Cited by 400 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 53 of its 119 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

53 of 119 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024MASAI: Modular Architecture for Software-engineering AI Agents17 Jun
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering6 May
  3. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repair25 Mar

Other references · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  11. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  12. 2021Measuring Coding Challenge Competence With APPScited by 879
38 more
  1. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  2. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  4. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  5. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
  6. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  7. 2020Unsupervised Translation of Programming Languagescited by 262
  8. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  9. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 191
  10. 2018SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repaircited by 144
  11. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  12. 2023Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPTcited by 125
  13. 2022Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproductioncited by 110
  14. 2018Mapping Language to Code in Programmatic Contextcited by 104
  15. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 102
  16. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 89
  17. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  18. 2021CURE: Code-Aware Neural Machine Translation for Automatic Program Repaircited by 85
  19. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  20. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  21. 2022Less Training, More Repairing Please: Revisiting Automated Program Repair via Zero-shot Learningcited by 76
  22. 2023Large Language Models for Test-Free Fault Localizationcited by 67
  23. 2019TBar: Revisiting Template-based Automated Program Repaircited by 64
  24. 2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
  25. 2021A Syntax-Guided Edit Decoder for Neural Program Repaircited by 57
  26. 2023A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localizationcited by 50
  27. 2024AgentFL: Scaling LLM-based Fault Localization to Project-Level Contextcited by 47
  28. 2023Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisionscited by 41
  29. 2024A Systematic Literature Review on Large Language Models for Automated Program Repaircited by 37
  30. 2023Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTcited by 36
  31. 2024CoverUp: Effective High Coverage Test Generation for Pythoncited by 36
  32. 2023Large Language Models in Fault Localisationcited by 28
  33. 2018Practical Program Repair via Bytecode Mutationcited by 19
  34. 2024Fuzzing BusyBox: Leveraging LLM and Crash Reuse for Embedded Bug Unearthingcited by 15
  35. 2024ACFIX: Guiding LLMs with Mined Common RBAC Practices for Context-Aware Repair of Access Control Vulnerabilities in Smart Contractscited by 15
  36. 2024CigaR: Cost-efficient Program Repair with LLMscited by 14
  37. 2023Programming-by-Demonstration for Long-Horizon Robot Taskscited by 9
  38. 2016An Analysis of the Search Spaces for Generate and Validate Patch Generation Systemscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 151; 2026: 209.

400 papers.

Sort:

By area:

  1. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  4. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  5. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  6. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  8. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  9. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 223
  10. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Taskscited by 221
  11. 2024ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discoverycited by 208
  12. 2025Survey on Evaluation of LLM-based Agentscited by 190
38 more
  1. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  2. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 165
  3. 2026MiMo-V2-Flash Technical Reportcited by 163
  4. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 150
  5. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  6. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  7. 2025MLGym: A New Framework and Benchmark for Advancing AI Research Agentscited by 111
  8. 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 106
  9. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 91
  10. 2024RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graphcited by 88
  11. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  12. 2025Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectoriescited by 75
  13. 2024SpecRover: Code Intent Extraction via LLMscited by 71
  14. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Taskscited by 69
  15. 2024MarsCode Agent: AI-native Automated Bug Fixingcited by 68
  16. 2025Paper2Code: Automating Code Generation from Scientific Papers in Machine Learningcited by 67
  17. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  18. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  19. 2025SEAL: Steerable Reasoning Calibration of Large Language Models for Freecited by 65
  20. 2025SWE-Exp: Experience-Driven Software Issue Resolutioncited by 64
  21. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  22. 2024HyperAgent: Generalist Software Engineering Agents to Solve Coding Tasks at Scalecited by 61
  23. 2024SelfCodeAlign: Self-Alignment for Code Generationcited by 61
  24. 2026Qwen3-Coder-Next Technical Reportcited by 60
  25. 2025LLM Agents for Education: Advances and Applicationscited by 59
  26. 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvementcited by 57
  27. 2026EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoningcited by 52
  28. 2025SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarioscited by 49
  29. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  30. 2025LLM Agents Making Agent Toolscited by 45
  31. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 45
  32. 2025Reducing Cost of LLM Agents with Trajectory Reductioncited by 44
  33. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 44
  34. 2024VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulationcited by 43
  35. 2025OrcaLoca: An LLM Agent Framework for Software Issue Localizationcited by 43
  36. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  37. 2024Human-In-the-Loop Software Development Agentscited by 42
  38. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 41

50 most cited of 400 All 400 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities
  2. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
  3. 2026Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
  4. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System
  5. 2025A Survey on Code Generation with LLM-based Agents
  6. 2024CodePori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
  7. 2026AgentForge: Execution-Grounded Multi-Agent LLM Framework for Autonomous Software Engineering
  8. 2025Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development