mlsanityData to 13 Oct 2026

CodeAlignBench: Assessing Code Generation Models on Developer-Preferred Code Adjustments

arXiv 2510.27565 ↗First posted 31 Oct 2025cs.SE cs.AI cs.HCcited by 0

Not on the map. Why?

Abstract (arXiv)

As large language models become increasingly capable of generating code, evaluating their performance remains a complex and evolving challenge. Existing benchmarks primarily focus on functional correctness, overlooking the diversity of real-world coding tasks and developer expectations. To this end, we introduce a multi-language benchmark that evaluates LLM instruction-following capabilities and is extensible to operate on any set of standalone coding problems. Our benchmark evaluates instruction following in two key settings: adherence to pre-defined constraints specified with the initial problem, and the ability to perform refinements based on follow-up instructions. For this paper's analysis, we empirically evaluated our benchmarking pipeline with programming tasks from LiveBench, that are also automatically translated from Python into Java and JavaScript. Our automated benchmark reveals that models exhibit differing levels of performance across multiple dimensions of instruction-following. Our benchmarking pipeline provides a more comprehensive evaluation of code generation models, highlighting their strengths and limitations across languages and generation goals.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

21 of 25 references.

References · 21

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,205
  2. 2021Program Synthesis with Large Language Modelscited by 3,573
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,274
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
  5. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
  7. 2021Measuring Coding Challenge Competence With APPScited by 891
  8. 2024A Survey on Large Language Models for Code Generationcited by 656
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 533
  10. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 395
  11. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 350
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 297
9 more
  1. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 250
  2. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 167
  3. 2024DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositoriescited by 71
  4. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  5. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 56
  6. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 32
  7. 2024From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Modelscited by 27
  8. 2025M-IFEval: Multilingual Instruction-Following Evaluationcited by 17
  9. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 15

Built on by

Not cited yet.