mlsanityData to 7 Oct 2026

MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues

arXiv 2402.14762 ↗First posted 22 Feb 2024cs.CL cs.AIcited by 247

Not on the map. Why?

Cited by 30 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The advent of Large Language Models (LLMs) has drastically enhanced dialogue systems. However, comprehensively evaluating the dialogue abilities of LLMs remains a challenge. Previous benchmarks have primarily focused on single-turn dialogues or provided coarse-grained and incomplete assessments of multi-turn dialogues, overlooking the complexity and fine-grained nuances of real-life dialogues. To address this issue, we introduce MT-Bench-101, specifically designed to evaluate the fine-grained abilities of LLMs in multi-turn dialogues. By conducting a detailed analysis of real multi-turn dialogue data, we construct a three-tier hierarchical ability taxonomy comprising 4208 turns across 1388 multi-turn dialogues in 13 distinct tasks. We then evaluate 21 popular LLMs based on MT-Bench-101, conducting comprehensive analyses from both ability and task perspectives and observing differing trends in LLMs performance across dialogue turns within various tasks. Further analysis indicates that neither utilizing common alignment techniques nor chat-specific designs has led to obvious enhancements in the multi-turn abilities of LLMs. Extensive case studies suggest that our designed tasks accurately assess the corresponding multi-turn abilities. The data and code are available at urlhttps://github.com/mtbench101/mt-bench-101.

Key facts. Cited by 247 papers in the map (data through 7 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 22 Feb 2024.

Builds on

29 of 54 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  8. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
17 more
  1. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  2. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  3. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  4. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  5. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  6. 2023The False Promise of Imitating Proprietary LLMscited by 254
  7. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  8. 2023Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMscited by 62
  9. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  10. 2023OWL: A Large Language Model for IT Operationscited by 43
  11. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!cited by 36
  12. 2020Large-Scale Object Detection in the Wild from Imbalanced Multi-Labelscited by 35
  13. 2023BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialoguescited by 34
  14. 2021GAIA: A Transfer Learning System of Object Detection that Fits Your Needscited by 27
  15. 2023Parrot: Enhancing Multi-Turn Instruction Following for Large Language Modelscited by 15
  16. 2018Solution for Large-Scale Hierarchical Object Detection Datasets with Incomplete Annotation and Data Imbalancecited by 7
  17. 2020Which Kind Is Better in Open-domain Multi-turn Dialog,Hierarchical or Non-hierarchical Models? An Empirical Studycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 123; 2026: 93.

247 papers.

Sort:

By area:

  1. 2025Why Do Multi-Agent LLM Systems Fail?cited by 559
  2. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  3. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  5. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  7. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  8. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  9. 2024GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Modelscited by 42
  10. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  11. 2025Know You First and Be You Better: Modeling Human-Like User Simulators via Implicit Profilescited by 38
  12. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
38 more
  1. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  2. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  3. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  4. 2024Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Modelscited by 28
  5. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  6. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 28
  7. 2024MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactionscited by 26
  8. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  9. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  10. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  11. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  12. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 25
  13. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  14. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
  15. 2025SafeDialBench: A Fine-Grained Safety Evaluation Benchmark for Large Language Models in Multi-Turn Dialogues with Diverse Jailbreak Attackscited by 23
  16. 2025On Robustness and Reliability of Benchmark-Based Evaluation of LLMscited by 23
  17. 2024Synthesizing Post-Training Data for LLMs through Multi-Agent Simulationcited by 22
  18. 2025Condor: Enhance LLM Alignment with Knowledge-Driven Data Synthesis and Refinementcited by 17
  19. 2025SOTOPIA-Omega: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agentscited by 16
  20. 2024A User-Centric Multi-Intent Benchmark for Evaluating Large Language Modelscited by 15
  21. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  22. 2025VocalBench: Benchmarking the Vocal Conversational Abilities for Speech Interaction Modelscited by 15
  23. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Modelscited by 14
  24. 2025RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decodingcited by 14
  25. 2025Can Language Models Follow Multiple Turns of Entangled Instructions?cited by 14
  26. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 14
  27. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  28. 2024GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Modelscited by 13
  29. 2024E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoningcited by 13
  30. 2025Learning Safety Constraints for Large Language Modelscited by 13
  31. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
  32. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testingcited by 13
  33. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  34. 2024LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluationcited by 12
  35. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  36. 2025Adaptive Fault Tolerance Mechanisms of Large Language Models in Cloud Computing Environmentscited by 12
  37. 2025MIRAGE: A Benchmark for Multimodal Information-Seeking and Reasoning in Agricultural Expert-Guided Conversationscited by 12
  38. 2026Confidence Estimation for LLMs in Multi-turn Interactionscited by 12

50 most cited of 247 All 247 (JSON)