mlsanityData to 14 Oct 2026

MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation

arXiv 2505.17123 ↗First posted 21 May 2025cs.CLcited by 15

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have shown promising results in complex reasoning tasks. However, current evaluations predominantly focus on single-turn reasoning scenarios, leaving interactive tasks largely unexplored. We attribute it to the absence of comprehensive datasets and scalable automatic evaluation protocols. To fill these gaps, we present MTR-Bench for LLMs' Multi-Turn Reasoning evaluation. Comprising 4 classes, 40 tasks, and 3600 instances, MTR-Bench covers diverse reasoning capabilities, fine-grained difficulty granularity, and necessitates multi-turn interactions with the environments. Moreover, MTR-Bench features fully-automated framework spanning both dataset constructions and model evaluations, which enables scalable assessment without human interventions. Extensive experiments reveal that even the cutting-edge reasoning models fall short of multi-turn, interactive reasoning tasks. And the further analysis upon these results brings valuable insights for future research in interactive AI systems.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 56 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

15 of 56 references.

References · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2024OpenAI o1 System Cardcited by 1,244
  10. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  11. 2025CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratingscited by 115
  12. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
3 more
  1. 2024Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBenchcited by 27
  2. 2025SPIN-Bench: How Well Do LLMs Plan Strategically and Reason Socially?cited by 24
  3. 2024A LLM Benchmark based on the Minecraft Builder Dialog Agent Taskcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agentscited by 11
  2. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  3. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
  4. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
  5. 2026FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoningcited by 4
  6. 2025Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Codecited by 3
  7. 2025WebGen-V Bench: Structured Representation for Enhancing Visual Design in LLM-based Web Generation and Evaluationcited by 3
  8. 2026EMSDialog: Synthetic Multi-person Emergency Medical Service Dialogue Generation from Electronic Patient Care Reports via Multi-LLM Agentscited by 2
  9. 2025What to Ask Next? Probing the Imaginative Reasoning of LLMs with TurtleSoup Puzzlescited by 0
  10. 2026Quantifying Cross-Query Contradictions in Multi-Query LLM Reasoningcited by 0
  11. 2026Do LLMs Capture Embodied Cognition and Cultural Variation? Cross-Linguistic Evidence from Demonstrativescited by 0
  12. 2026Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Gamescited by 0
3 more
  1. 2026ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Managementcited by 0
  2. 2026Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoningcited by 0
  3. 2026Clarification Is Not Correction: LLMs Fail to Let Gocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models
  2. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs
  3. 2025Multi-turn Training with Basic Human Feedback Helps Little on LLM Reasoning
  4. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialogues
  5. 2025MARS-Bench: A Multi-turn Athletic Real-world Scenario Benchmark for Dialogue Evaluation
  6. 2025mSCoRe: a Multilingual and Scalable Benchmark for Skill-based Commonsense Reasoning
  7. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insights
  8. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?