mlsanityData to 14 Oct 2026

FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models

arXiv 2310.20410 ↗First posted 31 Oct 2023cs.CLcited by 117

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The ability to follow instructions is crucial for Large Language Models (LLMs) to handle various real-world applications. Existing benchmarks primarily focus on evaluating pure response quality, rather than assessing whether the response follows constraints stated in the instruction. To fill this research gap, in this paper, we propose FollowBench, a Multi-level Fine-grained Constraints Following Benchmark for LLMs. FollowBench comprehensively includes five different types (i.e., Content, Situation, Style, Format, and Example) of fine-grained constraints. To enable a precise constraint following estimation on diverse difficulties, we introduce a Multi-level mechanism that incrementally adds a single constraint to the initial instruction at each increased level. To assess whether LLMs' outputs have satisfied every individual constraint, we propose to prompt strong LLMs with constraint-evolution paths to handle challenging open-ended instructions. By evaluating 13 closed-source and open-source popular LLMs on FollowBench, we highlight the weaknesses of LLMs in instruction following and point towards potential avenues for future work. The data and code are publicly available at https://github.com/YJiangcm/FollowBench.

Key facts. Cited by 117 papers in the map (data through 14 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

37 of 62 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
25 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  2. 2018Hierarchical Neural Story Generationcited by 1,193
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
  11. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  13. 2023Role-Play with Large Language Modelscited by 294
  14. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  15. 2023The False Promise of Imitating Proprietary LLMscited by 263
  16. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
  17. 2019SPoC: Search-based Pseudocode to Codecited by 176
  18. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 166
  19. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  20. 2021Document-Level Event Argument Extraction by Conditional Generationcited by 155
  21. 2020Learning from Task Descriptionscited by 78
  22. 2020Controlling Style in Generated Dialoguecited by 38
  23. 2022Controllable Text Generation with Language Constraintscited by 18
  24. 2021Style Control for Schema-Guided Natural Language Generationcited by 10
  25. 2022Replacing Language Model for Style Transfercited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 44; 2026: 50.

117 papers.

Sort:

By area:

  1. 2025Generalizing Verifiable Instruction Followingcited by 194
  2. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  4. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  5. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  6. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  7. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  8. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  9. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  10. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 54
  11. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  12. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
38 more
  1. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 49
  2. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  3. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  4. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  5. 2024SysBench: Can Large Language Models Follow System Messages?cited by 28
  6. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  7. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 27
  8. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
  9. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  10. 2025How Many Instructions Can LLMs Follow at Once?cited by 20
  11. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  12. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  13. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Modelscited by 13
  14. 2024Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leapscited by 13
  15. 2025Self-Steering Language Modelscited by 13
  16. 2025LiveTradeBench: Seeking Real-World Alpha with Large Language Modelscited by 13
  17. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 12
  18. 2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
  19. 2025WildIFEval: Instruction Following in the Wildcited by 11
  20. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  21. 2025ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interactioncited by 11
  22. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  23. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
  24. 2025Towards Better Instruction Following Retrieval Modelscited by 9
  25. 2025MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generationcited by 9
  26. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  27. 2025DNR Bench: Benchmarking Over-Reasoning in Reasoning LLMscited by 8
  28. 2025IF-VidCap: Can Video Caption Models Follow Instructions?cited by 7
  29. 2024Can I understand what I create? Self-Knowledge Evaluation of Large Language Modelscited by 6
  30. 2024Baichuan Alignment Technical Reportcited by 6
  31. 2025Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Followingcited by 6
  32. 2026From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generationcited by 6
  33. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 6
  34. 2025When Models Can't Follow: Testing Instruction Adherence Across 256 LLMscited by 5
  35. 2026When Is Enough Not Enough? Illusory Completion in Search Agentscited by 5
  36. 2026Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agentscited by 5
  37. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 5
  38. 2026S1-DeepResearch: Beyond Search, Toward Real-World Long-Horizon Research Agentscited by 5

50 most cited of 117 All 117 (JSON)