# COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis arXiv: https://arxiv.org/abs/2408.05006 ยท HTML: https://mlsanity.com/paper/2408.05006/ ## Key facts - Cited by 13 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 29 of its 50 references that are in the map. - First posted on arXiv 9 Aug 2024. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) - Synthetic data and self-improvement for LLMs / [Code and instruction data synthesis](https://mlsanity.com/synth/code-and-instruction-data-synthesis/) ## Builds on (most cited first, 29 of 29) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021) - [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024) - [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022) - [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024) - [Teaching Large Language Models to Self-Debug](https://mlsanity.com/paper/2304.05128/) (2023) - [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021) - [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024) - [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024) - [Is Self-Repair a Silver Bullet for Code Generation?](https://mlsanity.com/paper/2306.09896/) (2023) - [Advancing LLM Reasoning Generalists with Preference Trees](https://mlsanity.com/paper/2404.02078/) (2024) - [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023) - [Self-Edit: Fault-Aware Code Editor for Code Generation](https://mlsanity.com/paper/2305.04087/) (2023) - [Break-It-Fix-It: Unsupervised Learning for Program Repair](https://mlsanity.com/paper/2106.06600/) (2021) - [LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancement](https://mlsanity.com/paper/2403.15042/) (2024) - [TBar: Revisiting Template-based Automated Program Repair](https://mlsanity.com/paper/1903.08409/) (2019) - [RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program Repair](https://mlsanity.com/paper/2312.15698/) (2023) - [InstructCoder: Instruction Tuning Large Language Models for Code Editing](https://mlsanity.com/paper/2310.20329/) (2023) - [INTERVENOR: Prompting the Coding Ability of Large Language Models with the Interactive Chain of Repair](https://mlsanity.com/paper/2311.09868/) (2023) - [FixEval: Execution-based Evaluation of Program Fixes for Programming Problems](https://mlsanity.com/paper/2206.07796/) (2022) - [Review4Repair: Code Review Aided Automatic Program Repairing](https://mlsanity.com/paper/2010.01544/) (2020) - [Automatic Repair of Buggy If Conditions and Missing Preconditions with SMT](https://mlsanity.com/paper/1404.3186/) (2014) ## Built on by (most cited first, 13 of 13) - [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025) - [QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges](https://mlsanity.com/paper/2506.20008/) (2025) - [An LLM-enabled Multi-Agent Autonomous Mechatronics Design Framework](https://mlsanity.com/paper/2504.14681/) (2025) - [Teaching Large Reasoning Models Effective Reflection](https://mlsanity.com/paper/2601.12720/) (2026) - [Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?](https://mlsanity.com/paper/2604.17338/) (2026) - [From Context to Intent: Reasoning-Guided Function-Level Code Completion](https://mlsanity.com/paper/2508.09537/) (2025) - [Group Pattern Selection Optimization: Let LRMs Pick the Right Pattern for Reasoning](https://mlsanity.com/paper/2601.07238/) (2026) - [Anchored Self-Play for Code Repair](https://mlsanity.com/paper/2607.03523/) (2026) - [Why Stop at One Error? Benchmarking LLMs as Data Science Code Debuggers for Multi-Hop and Multi-Bug Errors](https://mlsanity.com/paper/2503.22388/) (2025) - [DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education](https://mlsanity.com/paper/2510.11076/) (2025) - [MapVerse: A Benchmark for Geospatial Question Answering on Diverse Real-World Maps](https://mlsanity.com/paper/2602.10518/) (2026) - [Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking](https://mlsanity.com/paper/2606.29088/) (2026) - [AI Assistants Overassist](https://mlsanity.com/paper/2607.21306/) (2026) ## Abstract (arXiv) Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST.