mlsanityData to 7 Oct 2026

InFoBench: Evaluating Instruction Following Ability in Large Language Models

arXiv 2401.03601 ↗First posted 7 Jan 2024cs.CL cs.AIcited by 128

Not on the map. Why?

Abstract (arXiv)

This paper introduces the Decomposed Requirements Following Ratio (DRFR), a new metric for evaluating Large Language Models' (LLMs) ability to follow instructions. Addressing a gap in current methodologies, DRFR breaks down complex instructions into simpler criteria, facilitating a detailed analysis of LLMs' compliance with various aspects of tasks. Alongside this metric, we present InFoBench, a benchmark comprising 500 diverse instructions and 2,250 decomposed questions across multiple constraint categories. Our experiments compare DRFR with traditional scoring methods and explore annotation sources, including human experts, crowd-sourced workers, and GPT-4. The findings demonstrate DRFR's higher reliability and the effectiveness of using GPT-4 as a cost-efficient annotator. The evaluation of several advanced LLMs using this framework reveals their strengths and areas needing improvement, particularly in complex instruction-following. This study contributes a novel metric and benchmark, offering insights for future LLM development and evaluation.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 29 of its 49 references that are in the map. First posted on arXiv 7 Jan 2024.

Builds on

29 of 49 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
17 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  11. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  13. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  14. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  15. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  16. 2021Alignment of Language Agentscited by 167
  17. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 61; 2026: 46.

128 papers.

Sort:

By area:

  1. 2025Generalizing Verifiable Instruction Followingcited by 178
  2. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  4. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  5. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  6. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  7. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  8. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 49
  9. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  10. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Trainingcited by 45
  11. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 44
  12. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 44
38 more
  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 41
  3. 2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?cited by 40
  4. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  5. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 34
  6. 2024Direct Judgement Preference Optimizationcited by 33
  7. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  9. 2025Speech-IFEval: Evaluating Instruction-Following and Quantifying Catastrophic Forgetting in Speech-Aware Language Modelscited by 27
  10. 2024SysBench: Can Large Language Models Follow System Messages?cited by 26
  11. 2024Do LLMs estimate uncertainty well in instruction-following?cited by 26
  12. 2024Do LLMs "know" internally when they follow instructions?cited by 25
  13. 2025OpenCharacter: Training Customizable Role-Playing LLMs with Large-Scale Synthetic Personascited by 25
  14. 2024LearnLM: Improving Gemini for Learningcited by 24
  15. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  16. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  17. 2025SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agentscited by 19
  18. 2025CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generationcited by 18
  19. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  20. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  21. 2025How Many Instructions Can LLMs Follow at Once?cited by 17
  22. 2024Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competitioncited by 16
  23. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 14
  24. 2024Innovative Thinking, Infinite Humor: Humor Research of Large Language Models through Structured Thought Leapscited by 12
  25. 2025From Drafts to Answers: Unlocking LLM Potential via Aggregation Fine-Tuningcited by 12
  26. 2025ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoningcited by 12
  27. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  28. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  29. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
  30. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 11
  31. 2025MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Modelscited by 10
  32. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  33. 2025Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?cited by 9
  34. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  35. 2024MMMT-IF: A Challenging Multimodal Multi-Turn Instruction Following Benchmarkcited by 8
  36. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  37. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8
  38. 2025IF-VidCap: Can Video Caption Models Follow Instructions?cited by 7

50 most cited of 128 All 128 (JSON)