# CODESIM: Multi-Agent Code Generation and Problem Solving through Simulation-Driven Planning and Debugging

arXiv: https://arxiv.org/abs/2502.05664 · HTML: https://mlsanity.com/paper/2502.05664/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 42 of its 60 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 42)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024)
- [Self-collaboration Code Generation via ChatGPT](https://mlsanity.com/paper/2304.07590/) (2023)
- [CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases](https://mlsanity.com/paper/1909.05378/) (2019)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [SantaCoder: don't reach for the stars!](https://mlsanity.com/paper/2301.03988/) (2023)
- [Retrieval Augmented Code Generation and Summarization](https://mlsanity.com/paper/2108.11601/) (2021)
- [Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step](https://mlsanity.com/paper/2402.16906/) (2024)
- [Task-Oriented Dialogue as Dataflow Synthesis](https://mlsanity.com/paper/2009.11423/) (2020)
- [Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://mlsanity.com/paper/2302.12822/) (2023)
- [Large Language Models as Analogical Reasoners](https://mlsanity.com/paper/2310.01714/) (2023)
- [Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering](https://mlsanity.com/paper/2401.08500/) (2024)

## Built on by (most cited first, 30 of 32)

- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [AgentConductor: Topology Evolution for Multi-Agent Competition-Level Code Generation](https://mlsanity.com/paper/2602.17100/) (2026)
- [Execution Guided Line-by-Line Code Generation](https://mlsanity.com/paper/2506.10948/) (2025)
- [Web Fraud Attacks Against LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2509.01211/) (2025)
- [PatchWorld: Gradient-Free Optimization of Executable World Models for Agent Environments](https://mlsanity.com/paper/2605.30880/) (2026)
- [CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback](https://mlsanity.com/paper/2507.22080/) (2025)
- [Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time](https://mlsanity.com/paper/2604.00917/) (2026)
- [Understanding and Bridging the Planner-Coder Gap: A Systematic Study on the Robustness of Multi-Agent Systems for Code Generation](https://mlsanity.com/paper/2510.10460/) (2025)
- [BioVerge: A Comprehensive Benchmark and Study of Self-Evaluating Agents for Biomedical Hypothesis Generation](https://mlsanity.com/paper/2511.08866/) (2025)
- [MapCoder-Lite: Distilling Multi-Agent Coding into a Single Small LLM](https://mlsanity.com/paper/2509.17489/) (2025)
- [LLM-Based Multi-Agent Systems for Code Generation: A Multi-Vocal Literature Review](https://mlsanity.com/paper/2604.16321/) (2026)
- [Position: agentic AI orchestration should be Bayes-consistent](https://mlsanity.com/paper/2605.00742/) (2026)
- [CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programming](https://mlsanity.com/paper/2605.24693/) (2026)
- [MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments](https://mlsanity.com/paper/2606.31966/) (2026)
- [The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents](https://mlsanity.com/paper/2603.24284/) (2026)
- [Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation](https://mlsanity.com/paper/2506.05623/) (2025)
- [Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents](https://mlsanity.com/paper/2510.14512/) (2025)
- [Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents](https://mlsanity.com/paper/2512.02812/) (2025)
- [CAM: A Causality-based Analysis Framework for Multi-Agent Code Generation Systems](https://mlsanity.com/paper/2602.02138/) (2026)
- [ARIADNE: Agentic Reward-Informed Adaptive Decision Exploration via Blackboard-Driven MCTS for Competitive Program Generation](https://mlsanity.com/paper/2605.02431/) (2026)
- [Poison with Style: A Practical Poisoning Attack on Code Large Language Models](https://mlsanity.com/paper/2605.27631/) (2026)
- [SwiftSolve: A Self-Iterative, Complexity-Aware Multi-Agent Framework for Competitive Programming](https://mlsanity.com/paper/2510.22626/) (2025)
- [CollabCoder: Plan-Code Co-Evolution via Collaborative Decision-Making for Efficient Code Generation](https://mlsanity.com/paper/2604.13946/) (2026)
- [SolidCoder: Bridging the Mental-Reality Gap in LLM Code Generation through Concrete Execution](https://mlsanity.com/paper/2604.19825/) (2026)
- [MAS-Algorithm: A Workflow for Solving Algorithmic Programming Problems with a Multi-Agent System](https://mlsanity.com/paper/2605.05949/) (2026)
- [Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows](https://mlsanity.com/paper/2605.14322/) (2026)
- [Agri-SAGE: Simulation-Grounded Multi-Agent LLM for Context-Aware Agricultural Advisory Generation](https://mlsanity.com/paper/2607.00454/) (2026)
- [Adversarial Review: Structured Disagreement for Grounded Agentic Code Review](https://mlsanity.com/paper/2608.18167/) (2026)
- [AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspace](https://mlsanity.com/paper/2608.23740/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have made significant strides in code generation and problem solving. Current approaches employ external tool-based iterative debuggers that use compiler or other tool-based runtime feedback to refine coarse programs generated by various methods. However, the effectiveness of these approaches heavily relies on the quality of the initial code generation, which remains an open challenge. In this paper, we introduce CodeSim, a novel multi-agent code generation framework that comprehensively addresses the stages of program synthesis-planning, coding, and debugging-through a human-like perception approach. As human verifies their understanding of any algorithms through visual simulation, CodeSim uniquely features a method of plan verification and internal debugging through the step-by-step simulation of input/output. Extensive experiments across seven challenging competitive problem-solving and program synthesis benchmarks demonstrate CodeSim's remarkable code generation capabilities. Our framework achieves new state-of-the-art (pass@1) results-(HumanEval 95.1%, MBPP 90.7%, APPS 22%, and CodeContests 29.1%). Furthermore, our method shows potential for even greater enhancement when cascaded with external debuggers. To facilitate further research and development in this area, we have open-sourced our framework in this link (https://kagnlp.github.io/codesim.github.io/).