mlsanityData to 7 Oct 2026

MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models

arXiv 2401.16745 ↗First posted 30 Jan 2024cs.CLcited by 78

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly relied upon for complex multi-turn conversations across diverse real-world applications. However, existing benchmarks predominantly focus on single-turn evaluations, overlooking the models' capabilities in multi-turn interactions. To address this gap, we introduce MT-Eval, a comprehensive benchmark designed to evaluate multi-turn conversational abilities. By analyzing human-LLM conversations, we categorize interaction patterns into four types: recollection, expansion, refinement, and follow-up. We construct multi-turn queries for each category either by augmenting existing datasets or by creating new examples with GPT-4 to avoid data leakage. To study the factors impacting multi-turn abilities, we create single-turn versions of the 1170 multi-turn queries and compare performance. Our evaluation of 11 well-known LLMs shows that while closed-source models generally surpass open-source ones, certain open-source models exceed GPT-3.5-Turbo in specific tasks. We observe significant performance degradation in multi-turn settings compared to single-turn settings in most models, which is not correlated with the models' fundamental capabilities. Moreover, we identify the distance to relevant content and susceptibility to error propagation as the key factors influencing multi-turn performance. MT-Eval is released publicly to encourage future research towards more robust conversational models.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

12 of 17 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  11. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  12. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Usecited by 100

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 32; 2026: 38.

78 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 117
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  4. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  5. 2025Training a Generally Curious Agentcited by 29
  6. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 25
  7. 2025SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agentscited by 21
  8. 2025A Personalized Conversational Benchmark: Towards Simulating Personalized Conversationscited by 16
  9. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 16
  10. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 14
  11. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  12. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
38 more
  1. 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
  2. 2025InfoQuest: Evaluating Multi-Turn Dialogue Agents for Open-Ended Conversations with Hidden Contextcited by 8
  3. 2025TeachLM: Post-Training LLMs for Education Using Authentic Learning Datacited by 8
  4. 2025Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Modelscited by 8
  5. 2024M2Lingual: Enhancing Multilingual, Multi-Turn Instruction Alignment in Large Language Modelscited by 7
  6. 2024MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systemscited by 7
  7. 2025GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketingcited by 7
  8. 2026MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Gamescited by 7
  9. 2025Substance over Style: Evaluating Proactive Conversational Coaching Agentscited by 6
  10. 2025B-score: Detecting biases in large language models using response historycited by 6
  11. 2025Time-To-Inconsistency: A Survival Analysis of Large Language Model Robustness to Adversarial Attackscited by 6
  12. 2025LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactionscited by 6
  13. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 6
  14. 2026Concept Influence: Leveraging Interpretability to Improve Performance and Efficiency in Training Data Attributioncited by 6
  15. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  16. 2026EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactionscited by 6
  17. 2024CURATe: Benchmarking Personalised Alignment of Conversational AI Assistantscited by 5
  18. 2026CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systemscited by 5
  19. 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 5
  20. 2025MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluationcited by 4
  21. 2025Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluationcited by 4
  22. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
  23. 2025Benchmarking Correctness and Security in Multi-Turn Code Generationcited by 4
  24. 2025Eliciting Behaviors in Multi-Turn Conversationscited by 4
  25. 2026Consistency of Large Reasoning Models Under Multi-Turn Attackscited by 4
  26. 2026A Neuropsychologically Grounded Evaluation of LLM Cognitive Abilitiescited by 4
  27. 2026Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systemscited by 4
  28. 2024Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Typescited by 3
  29. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 3
  30. 2025Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Promptingcited by 3
  31. 2026AI Alignment Breaks at the Edgecited by 3
  32. 2026Interactive Benchmarkscited by 3
  33. 2026When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interactioncited by 3
  34. 2025MedSyn: Enhancing Diagnostics with Human-AI Collaborationcited by 2
  35. 2025KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialoguescited by 2
  36. 2025Large Language Models Develop Novel Social Biases Through Adaptive Explorationcited by 2
  37. 2026Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideationcited by 2
  38. 2026DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflowscited by 2

50 most cited of 78 All 78 (JSON)