mlsanityData to 14 Oct 2026

Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following

arXiv 2410.15553 ↗First posted 21 Oct 2024cs.CLcited by 102

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated impressive capabilities in various tasks, including instruction following, which is crucial for aligning model outputs with user expectations. However, evaluating LLMs' ability to follow instructions remains challenging due to the complexity and subjectivity of human language. Current benchmarks primarily focus on single-turn, monolingual instructions, which do not adequately reflect the complexities of real-world applications that require handling multi-turn and multilingual interactions. To address this gap, we introduce Multi-IF, a new benchmark designed to assess LLMs' proficiency in following multi-turn and multilingual instructions. Multi-IF, which utilizes a hybrid framework combining LLM and human annotators, expands upon the IFEval by incorporating multi-turn sequences and translating the English prompts into another 7 languages, resulting in a dataset of 4,501 multilingual conversations, where each has three turns. Our evaluation of 14 state-of-the-art LLMs on Multi-IF reveals that it presents a significantly more challenging task than existing benchmarks. All the models tested showed a higher rate of failure in executing instructions correctly with each additional turn. For example, o1-preview drops from 0.877 at the first turn to 0.707 at the third turn in terms of average accuracy over all languages. Moreover, languages with non-Latin scripts (Hindi, Russian, and Chinese) generally exhibit higher error rates, suggesting potential limitations in the models' multilingual capabilities. We release Multi-IF prompts and the evaluation code base to encourage further research in this critical area.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

32 of 42 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,085
20 more
  1. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
  2. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  3. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  5. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  6. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  7. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 188
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  9. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  10. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  11. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  12. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  13. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  14. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 80
  15. 2023Evaluating Large Language Models on Controlled Generation Taskscited by 73
  16. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  17. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  18. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 66
  19. 2021Transferable Dialogue Systems and User Simulatorscited by 36
  20. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Languagecited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 50; 2026: 51.

102 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2025Qwen3-Omni Technical Reportcited by 497
  4. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 231
  5. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  7. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 49
  8. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
  9. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 35
  10. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 32
  11. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  12. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
38 more
  1. 2026Rethinking On-Policy Distillation of Large Language Models II: One Training Examplecited by 24
  2. 2025How Many Instructions Can LLMs Follow at Once?cited by 20
  3. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
  4. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  5. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 15
  6. 2025MulDimIF: A Multi-Dimensional Constraint Framework for Evaluating and Improving Instruction Following in Large Language Modelscited by 15
  7. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 15
  8. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?cited by 14
  9. 2025Marco-o1 v2: Towards Widening The Distillation Bottleneck for Reasoning Modelscited by 13
  10. 2025TextQuests: How Good are LLMs at Text-Based Video Games?cited by 12
  11. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  12. 2026ERNIE 5.0 Technical Reportcited by 11
  13. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  14. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMscited by 10
  15. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  16. 2026Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wildcited by 10
  17. 2026Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses?cited by 10
  18. 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
  19. 2025MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generationcited by 9
  20. 2025Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMscited by 7
  21. 2025FlowKV: Enhancing Multi-Turn Conversational Coherence in LLMs via Isolated Key-Value Cache Managementcited by 6
  22. 2026Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systemscited by 6
  23. 2026Sabi'a-4 Technical Reportcited by 6
  24. 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 6
  25. 2026Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scalecited by 6
  26. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 4
  27. 2025SEADialogues: A Multilingual Culturally Grounded Multi-turn Dialogue Dataset on Southeast Asian Languagescited by 4
  28. 2025Benchmarking Correctness and Security in Multi-Turn Code Generationcited by 4
  29. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  30. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  31. 2026Consistency of Large Reasoning Models Under Multi-Turn Attackscited by 4
  32. 2026Harness-IF: Evaluating Instruction Following Across Instruction Surfaces in Coding Agentscited by 4
  33. 2025Mind the Gap! Choice Independence in Using Multilingual LLMs for Persuasive Co-Writing Tasks in Different Languagescited by 3
  34. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLMcited by 3
  35. 2025Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capabilitycited by 3
  36. 2025BELLE: A Bi-Level Multi-Agent Reasoning Framework for Multi-Hop Question Answeringcited by 3
  37. 2025SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMscited by 3
  38. 2025On Evaluating LLM Alignment by Evaluating LLMs as Judgescited by 3

50 most cited of 102 All 102 (JSON)