mlsanityData to 7 Oct 2026

CFBench: A Comprehensive Constraints-Following Benchmark for LLMs

arXiv 2408.01122 ↗First posted 2 Aug 2024cs.CLcited by 65

Not on the map. Why?

Abstract (arXiv)

The adeptness of Large Language Models (LLMs) in comprehending and following natural language instructions is critical for their deployment in sophisticated real-world applications. Existing evaluations mainly focus on fragmented constraints or narrow scenarios, but they overlook the comprehensiveness and authenticity of constraints from the user's perspective. To bridge this gap, we propose CFBench, a large-scale Comprehensive Constraints Following Benchmark for LLMs, featuring 1,000 curated samples that cover more than 200 real-life scenarios and over 50 NLP tasks. CFBench meticulously compiles constraints from real-world instructions and constructs an innovative systematic framework for constraint types, which includes 10 primary categories and over 25 subcategories, and ensures each constraint is seamlessly integrated within the instructions. To make certain that the evaluation of LLM outputs aligns with user perceptions, we propose an advanced methodology that integrates multi-dimensional assessment criteria with requirement prioritization, covering various perspectives of constraints, instructions, and requirement fulfillment. Evaluating current leading LLMs on CFBench reveals substantial room for improvement in constraints following, and we further investigate influencing factors and enhancement strategies. The data and code are publicly available at https://github.com/PKU-Baichuan-MLSystemLab/CFBench

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 2 Aug 2024.

Builds on

32 of 42 references.

References · 32

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
20 more
  1. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  3. 2020Aligning AI With Shared Human Valuescited by 706
  4. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  7. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  8. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  9. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  10. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  11. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  12. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  13. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  14. 2020Learning from Task Descriptionscited by 78
  15. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 69
  16. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  17. 2022Controllable Text Generation with Language Constraintscited by 18
  18. 2023Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?cited by 16
  19. 2021Style Control for Schema-Guided Natural Language Generationcited by 10
  20. 2024Comprehensive Reassessment of Large-Scale Evaluation Outcomes in LLMs: A Multifaceted Statistical Approachcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 32; 2026: 24.

65 papers.

Sort:

By area:

  1. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 55
  2. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 44
  3. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  4. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  5. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  6. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 30
  7. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 28
  8. 2024SysBench: Can Large Language Models Follow System Messages?cited by 26
  9. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  10. 2025VeriPlan: Integrating Formal Verification and LLMs into End-User Planningcited by 19
  11. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 19
  12. 2025CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generationcited by 18
38 more
  1. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  2. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  3. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  4. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  5. 2025AIR: Complex Instruction Generation via Automatic Iterative Refinementcited by 10
  6. 2025WildIFEval: Instruction Following in the Wildcited by 10
  7. 2025Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modelingcited by 10
  8. 2025EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Modelscited by 9
  9. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
  10. 2025ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructionscited by 9
  11. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  12. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Followingcited by 8
  13. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  14. 2025IF-VidCap: Can Video Caption Models Follow Instructions?cited by 7
  15. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Followingcited by 7
  16. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMscited by 7
  17. 2024Baichuan Alignment Technical Reportcited by 6
  18. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  19. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
  20. 2025Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Followingcited by 6
  21. 2025Revisiting the Reliability of Language Models in Instruction-Followingcited by 6
  22. 2025R-ConstraintBench: Evaluating LLMs on NP-Complete Schedulingcited by 5
  23. 2025Prompt-Based One-Shot Exact Length-Controlled Generation with LLMscited by 4
  24. 2025When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Followingcited by 4
  25. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  26. 2025Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Followingcited by 3
  27. 2026SciIF: Benchmarking Scientific Instruction Following Towards Rigorous Scientific Intelligencecited by 3
  28. 2025Reverse Preference Optimization for Complex Instruction Followingcited by 2
  29. 2026SLBench: Evaluating How LLM Agents Follow Logical Relations in Skillscited by 2
  30. 2024Talking with Tables for Better LLM Factual Data Interactionscited by 1
  31. 2025UltraGen: Extremely Fine-grained Controllable Generation via Attribute Reconstruction and Global Preference Optimizationcited by 1
  32. 2025Format Inertia: A Failure Mechanism of LLMs in Medical Pre-Consultationcited by 1
  33. 2025GraphIF: Enhancing Multi-Turn Instruction Following for Large Language Models with Relation Graph Promptcited by 1
  34. 2026Precision over Diversity: High-Precision Reward Generalizes to Robust Instruction Followingcited by 1
  35. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillationcited by 1
  36. 2026CCR-Bench: A Comprehensive Benchmark for Evaluating LLMs on Complex Constraints, Control Flows, and Real-World Casescited by 1
  37. 2026SEIF: Self-Evolving Reinforcement Learning for Instruction Followingcited by 1
  38. 2026IFHierBench: Hierarchical Instruction Following for Large Language Modelscited by 1

50 most cited of 65 All 65 (JSON)