mlsanityData to 7 Oct 2026

A Survey on Code Generation with LLM-based Agents

arXiv 2508.00083 ↗First posted 31 Jul 2025cs.SE cs.AI cs.CLcited by 104

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Code generation agents powered by large language models (LLMs) are revolutionizing the software development paradigm. Distinct from previous code generation techniques, code generation agents are characterized by three core features. 1) Autonomy: the ability to independently manage the entire workflow, from task decomposition to coding and debugging. 2) Expanded task scope: capabilities that extend beyond generating code snippets to encompass the full software development lifecycle (SDLC). 3) Enhancement of engineering practicality: a shift in research emphasis from algorithmic innovation toward practical engineering challenges, such as system reliability, process management, and tool integration. This domain has recently witnessed rapid development and an explosion in research, demonstrating significant application potential. This paper presents a systematic survey of the field of LLM-based code generation agents. We trace the technology's developmental trajectory from its inception and systematically categorize its core techniques, including both single-agent and multi-agent architectures. Furthermore, this survey details the applications of LLM-based agents across the full SDLC, summarizes mainstream evaluation benchmarks and metrics, and catalogs representative tools. Finally, by analyzing the primary challenges, we identify and propose several foundational, long-term research directions for the future work of the field.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 2 of its 169 references that are in the map. First posted on arXiv 31 Jul 2025.

Builds on

2 of 169 references.

References · 2

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Improving Grounded Language Understanding in a Collaborative Environment by Interacting with Agents Through Help Feedbackcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 75.

104 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 72
  3. 2025CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel Optimizationcited by 42
  4. 2026Code as Agent Harnesscited by 40
  5. 2025CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignmentcited by 22
  6. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  7. 2025Retrieval-Augmented Code Generation: A Survey with Focus on Repository-Level Approachescited by 20
  8. 2025SWE-Effi: Re-Evaluating Software AI Agent System Effectiveness Under Resource Constraintscited by 17
  9. 2026Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verificationcited by 16
  10. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  11. 2025Magentic Marketplace: An Open-Source Environment for Studying Agentic Marketscited by 15
  12. 2025DeepCode: Open Agentic Codingcited by 14
38 more
  1. 2025Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Modelcited by 13
  2. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  3. 2025CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Codingcited by 11
  4. 2026Controlled Self-Evolution for Algorithmic Code Optimizationcited by 11
  5. 2026KernelSkill: A Multi-Agent Framework for GPU Kernel Optimizationcited by 9
  6. 2025Explainable and Fine-Grained Safeguarding of LLM Multi-Agent Systems via Bi-Level Graph Anomaly Detectioncited by 8
  7. 2026ChipBench: A Next-Step Benchmark for Evaluating LLM Performance in AI-Aided Chip Designcited by 7
  8. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
  9. 2025Automatically Generating Web Applications from Requirements Via Multi-Agent Test-Driven Developmentcited by 5
  10. 2025Multi-Agent Code-Orchestrated Generation for Reliable Infrastructure-as-Codecited by 5
  11. 2025CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournamentscited by 5
  12. 2026Large Language Model Agent for User-friendly Chemical Process Simulationscited by 5
  13. 2026Codified Context: Infrastructure for AI Agents in a Complex Codebasecited by 5
  14. 2026Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomycited by 5
  15. 2026ICAE-Bench: Evaluating Coding Agents as Interactive Project Builderscited by 5
  16. 2025VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specificationscited by 4
  17. 2025AI Code in the Wild: Measuring Security Risks and Ecosystem Shifts of AI-Generated Code in Modern Softwarecited by 4
  18. 2025SweRank+: Multilingual, Multi-Turn Code Ranking for Software Issue Localizationcited by 4
  19. 2026EComStage: Stage-wise and Orientation-specific Benchmarking for Large Language Models in E-commercecited by 4
  20. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problemcited by 4
  21. 2026KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardwarecited by 4
  22. 2026Your Code Agent Can Grow Alongside You with Structured Memorycited by 4
  23. 2026FAMA: Failure-Aware Meta-Agentic Framework for Open-Source LLMs in Interactive Tool Use Environmentscited by 4
  24. 2026RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgradescited by 4
  25. 2025Reasoning or Rambling? Exploring the Effect of Thinking on Agent Persuasioncited by 3
  26. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 3
  27. 2026What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understandingcited by 3
  28. 2026Beyond Quantity: Trajectory Diversity Scaling for Code Agentscited by 3
  29. 2026Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Timecited by 3
  30. 2026SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineeringcited by 3
  31. 2025textttReMind: Understanding Deductive Code Reasoning in LLMscited by 2
  32. 2025Evaluating Software Process Models for Multi-Agent Class-Level Code Generationcited by 2
  33. 2025KernelBand: Steering LLM-based Kernel Optimization via Hardware-Aware Multi-Armed Banditscited by 2
  34. 2025CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agentscited by 2
  35. 2026MagicAgent: Towards Generalized Agent Planningcited by 2
  36. 2026VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?cited by 2
  37. 2026Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolutioncited by 2
  38. 2026El Agente Forjador: Task-Driven Agent Generation for Quantum Simulationcited by 2

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
  2. 2025Towards Advancing Code Generation with Large Language Models: A Research Roadmap
  3. 2026Developing LLM-based Multi-Agent Systems in Software Engineering: A Mixed-Method Experience Report
  4. 2025AdaCoder: An Adaptive Planning and Multi-Agent Framework for Function-Level Code Generation
  5. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation
  6. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
  7. 2024Self-Organized Agents: A LLM Multi-Agent Framework toward Ultra Large-Scale Code Generation and Optimization
  8. 2026LLM-Based Agentic Systems for Software Engineering: Challenges and Opportunities