mlsanityData to 14 Oct 2026

CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation

arXiv 2503.22688 ↗First posted 5 Mar 2025cs.SE cs.AI cs.PLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated exceptional performance in code generation tasks and have become indispensable programming assistants for developers. However, existing code generation benchmarks primarily assess the functional correctness of code generated by LLMs in single-turn interactions. They offer limited insight into LLMs' abilities to generate code that strictly follows users' instructions in multi-turn interaction scenarios. In this paper, we introduce CodeIF-Bench, a benchmark for evaluating the instruction-following capabilities of LLMs in interactive code generation. Specifically, CodeIF-Bench incorporates nine types of verifiable instructions aligned with the real-world software development requirements, which can be independently and objectively validated through specified test cases, facilitating the evaluation of instruction-following capability in multi-turn interactions. In both Static Conversation and Dynamic Conversation settings, we evaluate the performance of 6 state-of-the-art LLMs and summarize the important factors, additional repository context and gradually increasing interaction history influencing the instruction-following ability of LLMs in multi-turn interactions. Furthermore, we identify the potential direction for improvement: context management. The code and data are available at hrefhttps://github.com/zhu-zhu-ding/CodeIF-Benchhttps://github.com/zhu-zhu-ding/CodeIF-Bench.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

32 of 46 references.

References · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2021Measuring Coding Challenge Competence With APPScited by 902
  11. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  12. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
20 more
  1. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  2. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 331
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  4. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  5. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 247
  6. 2022Do Users Write More Insecure Code with AI Assistants?cited by 192
  7. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  8. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  9. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  10. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  11. 2022Automating Code Review Activities by Large-Scale Pre-trainingcited by 98
  12. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
  13. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 72
  14. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  15. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 60
  16. 2023Can It Edit? Evaluating the Ability of Large Language Models to Follow Code Editing Instructionscited by 48
  17. 2024A Taxonomy for Human-LLM Interaction Modes: An Initial Explorationcited by 38
  18. 2025CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generationcited by 19
  19. 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedbackcited by 11
  20. 2024Inductive-Deductive Strategy Reuse for Multi-Turn Instructional Dialoguescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 7.

15 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  2. 2026SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Taskscited by 30
  3. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
  4. 2025KAT-Coder Technical Reportcited by 7
  5. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 7
  6. 2025OSS-Bench: Benchmark Generator for Coding LLMscited by 6
  7. 2026ClarifyCodeBench: Evaluating LLMs on Clarifying Ambiguous Requirements for Code Generationcited by 5
  8. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 4
  9. 2026Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agentscited by 4
  10. 2026CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Developmentcited by 2
  11. 2025CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustmentscited by 0
  12. 2025PACIFIC: a framework for generating benchmarks to check Precise Automatically Checked Instruction Following In Codecited by 0
3 more
  1. 2026Failing to Explore: Language Models on Interactive Taskscited by 0
  2. 2026EduPluginBench: Executable Assurance for AI-Generated Educational Pluginscited by 0
  3. 2026MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Codingcited by 0