mlsanityData to 14 Oct 2026

Benchmarking Complex Instruction-Following with Multiple Constraints Composition

arXiv 2407.03978 ↗First posted 4 Jul 2024cs.CL cs.AIcited by 152

Not on the map. Why?

Cited by 29 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to deal with complex human instructions in real-world scenarios. Therefore, how to evaluate the ability of complex instruction-following of LLMs has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints, which is an indispensable constituent in complex instructions. To this end, we propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints. We propose a hierarchical taxonomy for complex instructions, including 4 constraint types, 19 constraint dimensions, and 4 composition types, and manually collect a high-quality dataset accordingly. To make the evaluation reliable, we augment LLM-based evaluators with rules to effectively verify whether generated texts can satisfy each constraint and composition. Furthermore, we obtain the final evaluation score based on the dependency structure determined by different composition types. ComplexBench identifies significant deficiencies in existing LLMs when dealing with complex instructions with multiple constraints composition.

Key facts. Cited by 152 papers in the map (data through 14 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 4 Jul 2024.

Builds on

41 of 53 references.

References · 41

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023Qwen Technical Reportcited by 3,344
  7. 2023Mistral 7Bcited by 3,164
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
29 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  3. 2024InternLM2 Technical Reportcited by 397
  4. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  5. 2018Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfercited by 211
  6. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 183
  7. 2017Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memorycited by 167
  8. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  9. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  10. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  11. 2019Measuring Compositionality in Representation Learningcited by 84
  12. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 80
  13. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  14. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  15. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  16. 2017MojiTalk: Generating Emotional Responses at Scalecited by 64
  17. 2017Neural AMR: Sequence-to-Sequence Models for Parsing and Generationcited by 56
  18. 2016Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversationcited by 35
  19. 2021StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfercited by 34
  20. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  21. 2023CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Modelscited by 34
  22. 2021Disentangled Sequence to Sequence Learning for Compositional Generalizationcited by 30
  23. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructionscited by 27
  24. 2021Improving Compositional Generalization with Self-Training for Data-to-Text Generationcited by 19
  25. 2022Why is constrained neural language generation particularly challenging?cited by 19
  26. 2024Chain-of-Instructions: Compositional Instruction Tuning on Large Language Modelscited by 9
  27. 2023DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answeringcited by 7
  28. 2024Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generationcited by 5
  29. 2023Compositional Generalization for Data-to-Text Generationcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 70; 2026: 65.

152 papers.

Sort:

By area:

  1. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  2. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  3. 2024CFBench: A Comprehensive Constraints-Following Benchmark for LLMscited by 67
  4. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  5. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 53
  6. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  7. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 49
  8. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 36
  9. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  10. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 33
  11. 2024IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimizationcited by 31
  12. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 29
38 more
  1. 2024SysBench: Can Large Language Models Follow System Messages?cited by 28
  2. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  3. 2024RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarioscited by 27
  4. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  5. 2024LearnLM: Improving Gemini for Learningcited by 25
  6. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 24
  7. 2025NavBench: Probing Multimodal Large Language Models for Embodied Navigationcited by 24
  8. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  9. 2025SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraintscited by 23
  10. 2025SOP-Bench: Complex Industrial SOPs for Evaluating LLM Agentscited by 23
  11. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 21
  12. 2025How Many Instructions Can LLMs Follow at Once?cited by 20
  13. 2026RubricEval: A Rubric-Level Meta-Evaluation Benchmark for LLM Judges in Instruction Followingcited by 20
  14. 2025MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Modelscited by 15
  15. 2025AssetOpsBench: Benchmarking AI Agents for Task Automation in Industrial Asset Operations and Maintenancecited by 15
  16. 2025GuideBench: Benchmarking Domain-Oriented Guideline Following for LLM Agentscited by 14
  17. 2025LLMs can be easily Confused by Instructional Distractionscited by 13
  18. 2025A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agentscited by 13
  19. 2025MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Trainingcited by 12
  20. 2025Order Matters: Investigate the Position Bias in Multi-constraint Instruction Followingcited by 12
  21. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
  22. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 12
  23. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  24. 2025RLMR: Reinforcement Learning with Mixed Rewards for Creative Writingcited by 11
  25. 2025AIR: Complex Instruction Generation via Automatic Iterative Refinementcited by 10
  26. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  27. 2025Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?cited by 10
  28. 2025ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructionscited by 10
  29. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 10
  30. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 10
  31. 2026Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skillscited by 10
  32. 2024Toward General Instruction-Following Alignment for Retrieval-Augmented Generationcited by 9
  33. 2025EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Modelscited by 9
  34. 2025Routine: A Structural Planning Framework for LLM Agent System in Enterprisecited by 9
  35. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
  36. 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarioscited by 9
  37. 2024Divide-Verify-Refine: Can LLMs Self-Align with Complex Instructions?cited by 8
  38. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Followingcited by 8

50 most cited of 152 All 152 (JSON)