mlsanityData to 7 Oct 2026

CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges

arXiv 2401.07339 ↗First posted 14 Jan 2024cs.SEcited by 278

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Large Language Models (LLMs) have shown promise in automated code generation but typically excel only in simpler tasks such as generating standalone code units. Real-world software development, however, often involves complex code repositories (named repo) with complex dependencies and extensive documentation. To fill this gap, our research pivots towards evaluating LLMs in a more realistic setting -- real-world repo-level code generation. We introduce CodeAgentBench, a manually curated benchmark for repo-level code generation. This benchmark comprises five high-quality Python projects, encompassing a total of 101 samples. We assess nine leading LLMs on repo-level tasks and observe a decline in their performance. To tackle this, we present CodeAgent, a novel LLM-based agent framework that employs external tools for effective repo-level code generation. CodeAgent integrates five programming tools, enabling interaction with software artifacts for information retrieval, code symbol navigation, and code testing. We implement four agent strategies to optimize these tools' usage. Our experiments on CodeAgentBench show that CodeAgent enhances LLM performance significantly, with improvements ranging from 18.1% to 250%. Further tests on the HumanEval benchmark confirm CodeAgent's adaptability and efficacy across various code generation tasks. Notably, CodeAgent outperforms commercial products like Github Copilot, showcasing superior accuracy and efficiency. These results demonstrate CodeAgent's robust capabilities in code generation, highlighting its potential for real-world repo-level coding challenges.

Key facts. Cited by 278 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 30 of its 43 references that are in the map. First posted on arXiv 14 Jan 2024.

Builds on

30 of 43 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
18 more
  1. 2023Teaching Large Language Models to Self-Debugcited by 963
  2. 2021Measuring Coding Challenge Competence With APPScited by 879
  3. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  4. 2023StarCoder: may the source be with you!cited by 790
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  7. 2022CodeT: Code Generation with Generated Testscited by 369
  8. 2023InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedbackcited by 229
  9. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 166
  10. 2018Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflowcited by 165
  11. 2022Multi-lingual Evaluation of Code Generation Modelscited by 158
  12. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 150
  13. 2015Yara Parser: A Fast and Accurate Dependency Parsercited by 110
  14. 2023Self-Edit: Fault-Aware Code Editor for Code Generationcited by 110
  15. 2018Mapping Language to Code in Programmatic Contextcited by 104
  16. 2022CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generationcited by 87
  17. 2023Towards A Unified Agent with Foundation Modelscited by 46
  18. 2024DevEval: Evaluating Code Generation in Practical Software Projectscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 42; 2025: 112; 2026: 124.

278 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Models for Code Generationcited by 646
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  3. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  4. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  5. 2024Retrieval-Augmented Generation for AI-Generated Content: A Surveycited by 352
  6. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 243
  7. 2024Tool Learning with Large Language Models: A Surveycited by 221
  8. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  9. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 173
  10. 2025Towards a Science of Scaling Agent Systemscited by 125
  11. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 115
  12. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
38 more
  1. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  2. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  3. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  4. 2024Beyond Browsing: API-Based Web Agentscited by 77
  5. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  6. 2025SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agentscited by 66
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  8. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  9. 2024Agents in Software Engineering: Survey, Landscape, and Visioncited by 60
  10. 2025LLM Agents for Education: Advances and Applicationscited by 59
  11. 2024Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Explorationcited by 58
  12. 2025A Survey of LLM-based Deep Search Agents: Paradigm, Optimization, Evaluation, and Challengescited by 58
  13. 2024DA-Code: Agent Data Science Code Generation Benchmark for Large Language Modelscited by 57
  14. 2024Lingma SWE-GPT: An Open Development-Process-Centric Language Model for Automated Software Improvementcited by 57
  15. 2025Reinforcement Learning from Human Feedbackcited by 53
  16. 2025Training-Free Group Relative Policy Optimizationcited by 53
  17. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
  18. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  19. 2024From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debuggingcited by 46
  20. 2024AgileCoder: Dynamic Collaborative Agents for Software Development based on Agile Methodologycited by 45
  21. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  22. 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 44
  23. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  24. 2025SciReplicate-Bench: Benchmarking LLMs in Agent-driven Algorithmic Reproduction from Research Paperscited by 42
  25. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 42
  26. 2025Repo2Run: Automated Building Executable Environment for Code Repository at Scalecited by 39
  27. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  28. 2025Understanding the Information Propagation Effects of Communication Topologies in LLM-based Multi-Agent Systemscited by 39
  29. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  30. 2025SWE-QA: Can Language Models Answer Repository-level Code Questions?cited by 37
  31. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  32. 2024Benchmarking Agentic Workflow Generationcited by 36
  33. 2025ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verificationcited by 35
  34. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generationcited by 33
  35. 2025CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debuggingcited by 31
  36. 2024DynaSaur: Large Language Agents Beyond Predefined Actionscited by 30
  37. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Researchcited by 29
  38. 2024A Survey of Useful LLM Evaluationcited by 28

50 most cited of 278 All 278 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents
  2. 2025A Survey on Code Generation with LLM-based Agents
  3. 2023AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation
  4. 2025AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents
  5. 2025DocAgent: A Multi-Agent System for Automated Code Documentation Generation
  6. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents
  7. 2026E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
  8. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging