# AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

arXiv: https://arxiv.org/abs/2312.13010 · HTML: https://mlsanity.com/paper/2312.13010/

## Key facts

- Cited by 31 papers in the map (data through 7 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 36 of its 41 references that are in the map.
- First posted on arXiv 20 Dec 2023.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 36)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors](https://mlsanity.com/paper/2308.10848/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [CodeT5+: Open Code Large Language Models for Code Understanding and Generation](https://mlsanity.com/paper/2305.07922/) (2023)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)
- [CodeGen2: Lessons for Training LLMs on Programming and Natural Languages](https://mlsanity.com/paper/2305.02309/) (2023)
- [Self-planning Code Generation with Large Language Models](https://mlsanity.com/paper/2303.06689/) (2023)

## Built on by (most cited first, 30 of 31)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL](https://mlsanity.com/paper/2508.13167/) (2025)
- [Do as We Do, Not as You Think: the Conformity of Large Language Models](https://mlsanity.com/paper/2501.13381/) (2025)
- [Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMs](https://mlsanity.com/paper/2502.19411/) (2025)
- [Code as Agent Harness](https://mlsanity.com/paper/2605.18747/) (2026)
- [MAGE: A Multi-Agent Engine for Automated RTL Code Generation](https://mlsanity.com/paper/2412.07822/) (2024)
- [ReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning](https://mlsanity.com/paper/2506.09513/) (2025)
- [LLM-based Agentic Reasoning Frameworks: A Survey from Methods to Scenarios](https://mlsanity.com/paper/2508.17692/) (2025)
- [Revisit Self-Debugging with Self-Generated Tests for Code Generation](https://mlsanity.com/paper/2501.12793/) (2025)
- [LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering](https://mlsanity.com/paper/2511.13998/) (2025)
- [A Human-Like Reasoning Framework for Multi-Phases Planning Task with Large Language Models](https://mlsanity.com/paper/2405.18208/) (2024)
- [SOP-Agent: Empower General Purpose AI Agent with Domain-Specific SOPs](https://mlsanity.com/paper/2501.09316/) (2025)
- [SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering](https://mlsanity.com/paper/2502.06994/) (2025)
- [HoarePrompt: Structural Reasoning About Program Correctness in Natural Language](https://mlsanity.com/paper/2503.19599/) (2025)
- [Pragmatic Reasoning improves LLM Code Generation](https://mlsanity.com/paper/2502.15835/) (2025)
- [Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models](https://mlsanity.com/paper/2410.13413/) (2024)
- [RepoReasoner: Evaluating Repository-Level Code Reasoning Ability of Long-Context Language Models](https://mlsanity.com/paper/2607.25996/) (2026)
- [MAPLE: Multi-Agent Adaptive Planning with Long-Term Memory for Table Reasoning](https://mlsanity.com/paper/2506.05813/) (2025)
- [Can Github issues be solved with Tree Of Thoughts?](https://mlsanity.com/paper/2405.13057/) (2024)
- [MaCTG: Multi-Agent Collaborative Thought Graph for Automatic Programming](https://mlsanity.com/paper/2410.19245/) (2024)
- [Seed-CTS: Unleashing the Power of Tree Search for Superior Performance in Competitive Coding Tasks](https://mlsanity.com/paper/2412.12544/) (2024)
- [Thinking Before Running! Efficient Code Generation with Thorough Exploration and Optimal Refinement](https://mlsanity.com/paper/2502.17442/) (2024)
- [AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning](https://mlsanity.com/paper/2508.11995/) (2025)
- [TDD-Agent: Test-Driven Reasoning for Code Generation](https://mlsanity.com/paper/2608.16742/) (2026)
- [Lessons Learned: A Multi-Agent Framework for Code LLMs to Learn and Improve](https://mlsanity.com/paper/2505.23946/) (2025)
- [Reasoning Distillation and Structural Alignment for Improved Code Generation](https://mlsanity.com/paper/2510.17598/) (2025)
- [SOCIA-Nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation](https://mlsanity.com/paper/2510.18551/) (2025)
- [DIANOIA: Diagnostic Decomposition and Joint Optimization for Multi-Agent Reasoning](https://mlsanity.com/paper/2602.08586/) (2026)
- [Selective Left-Shift: Turning Test-Time Compute and Difficulty-based Curation into Training Data for Low-Resource Code Generation](https://mlsanity.com/paper/2607.07748/) (2026)
- [Homogenization in Multi-Agent Systems](https://mlsanity.com/paper/2610.09824/) (2026)

## Abstract (arXiv)

The advancement of natural language processing (NLP) has been significantly boosted by the development of transformer-based large language models (LLMs). These models have revolutionized NLP tasks, particularly in code generation, aiding developers in creating software with enhanced efficiency. Despite their advancements, challenges in balancing code snippet generation with effective test case generation and execution persist. To address these issues, this paper introduces Multi-Agent Assistant Code Generation (AgentCoder), a novel solution comprising a multi-agent framework with specialized agents: the programmer agent, the test designer agent, and the test executor agent. During the coding procedure, the programmer agent will focus on the code generation and refinement based on the test executor agent's feedback. The test designer agent will generate test cases for the generated code, and the test executor agent will run the code with the test cases and write the feedback to the programmer. This collaborative system ensures robust code generation, surpassing the limitations of single-agent models and traditional methodologies. Our extensive experiments on 9 code generation models and 12 enhancement approaches showcase AgentCoder's superior performance over existing code generation models and prompt engineering techniques across various benchmarks. For example, AgentCoder (GPT-4) achieves 96.3% and 91.8% pass@1 in HumanEval and MBPP datasets with an overall token overhead of 56.9K and 66.3K, while state-of-the-art obtains only 90.2% and 78.9% pass@1 with an overall token overhead of 138.2K and 206.5K.