# CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation

arXiv: https://arxiv.org/abs/2504.21751 · HTML: https://mlsanity.com/paper/2504.21751/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 47 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 30 of 32)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation](https://mlsanity.com/paper/2211.11501/) (2022)
- [CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution](https://mlsanity.com/paper/2401.03065/) (2024)
- [Magicoder: Empowering Code Generation with OSS-Instruct](https://mlsanity.com/paper/2312.02120/) (2023)
- [InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback](https://mlsanity.com/paper/2306.14898/) (2023)
- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [MapCoder: Multi-Agent Code Generation for Competitive Problem Solving](https://mlsanity.com/paper/2405.11403/) (2024)
- [CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings](https://mlsanity.com/paper/2501.01257/) (2025)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories](https://mlsanity.com/paper/2404.00599/) (2024)
- [EffiBench: Benchmarking the Efficiency of Automatically Generated Code](https://mlsanity.com/paper/2402.02037/) (2024)
- [HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation](https://mlsanity.com/paper/2412.21199/) (2024)
- [DOMAINEVAL: An Auto-Constructed Benchmark for Multi-Domain Code Generation](https://mlsanity.com/paper/2408.13204/) (2024)
- [Agile Software Development Methods: Review and Analysis](https://mlsanity.com/paper/1709.08439/) (2017)

## Built on by (most cited first, 17 of 17)

- [SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks](https://mlsanity.com/paper/2603.24755/) (2026)
- [EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions](https://mlsanity.com/paper/2605.24110/) (2026)
- [ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions](https://mlsanity.com/paper/2603.00187/) (2026)
- [SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades](https://mlsanity.com/paper/2605.14415/) (2026)
- [When the Specification Emerges: Benchmarking Faithfulness Loss in Long-Horizon Coding Agents](https://mlsanity.com/paper/2603.17104/) (2026)
- [ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generation](https://mlsanity.com/paper/2607.00711/) (2026)
- [Context Learning for Multi-Agent Discussion](https://mlsanity.com/paper/2602.02350/) (2026)
- [On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows](https://mlsanity.com/paper/2605.06110/) (2026)
- [Repo0: Design-Driven Zero-to-All Code Generation](https://mlsanity.com/paper/2608.19854/) (2026)
- [Is Agent Code Less Maintainable Than Human Code?](https://mlsanity.com/paper/2606.21804/) (2026)
- [A-ProS: Towards Reliable Autonomous Programming Through Multi-Model Feedback](https://mlsanity.com/paper/2605.18073/) (2026)
- [StaminaBench: Stress-Testing Coding Agents over 100 Interaction Turns](https://mlsanity.com/paper/2606.19613/) (2026)
- [GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development](https://mlsanity.com/paper/2609.21293/) (2026)
- [Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification](https://mlsanity.com/paper/2511.00202/) (2025)
- [Strategies for Guiding LLMs to Use Software Design Patterns: A Case of Singleton](https://mlsanity.com/paper/2605.26898/) (2026)
- [CoSTAR: Data Synthesis-Driven Constraint-Aware COBOL Section Summarization for Legacy System Modernization](https://mlsanity.com/paper/2609.11332/) (2026)
- [Do Coding Agents Reuse Existing Code or Reinvent the Wheel?](https://mlsanity.com/paper/2609.35357/) (2026)

## Abstract (arXiv)

Modern software development demands code that is maintainable, testable, and scalable by organizing the implementation into modular components with iterative reuse of existing codes. We formalize this iterative, multi-turn paradigm as codeflow and introduce CodeFlowBench, the first benchmark designed to comprehensively evaluate LLMs' ability to perform codeflow - implementing new functionality by reusing existing functions over multiple turns. CodeFlowBench comprises two complementary components: CodeFlowBench-Comp, a core collection of 5,000+ competitive programming problems from Codeforces updated via an automated pipeline and CodeFlowBench-Repo, which is sourced from GitHub repositories to better reflect real-world scenarios. Furthermore, a novel evaluation framework featured dual assessment protocol and structural metrics derived from dependency trees is introduced. Extensive experiments reveal significant performance degradation in multi-turn codeflow scenarios. Furthermore, our in-depth analysis illustrates that model performance inversely correlates with dependency complexity. These findings not only highlight the critical challenges for supporting real-world workflows, but also establish CodeFlowBench as an essential tool for advancing code generation research.