mlsanityData to 7 Oct 2026

MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs

arXiv 2501.17399 ↗First posted 29 Jan 2025cs.CL cs.AIcited by 171

Not on the map. Why?

Cited by 40 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy.

Key facts. Cited by 171 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jan 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 66; 2026: 105.

171 papers.

Sort:

By area:

  1. 2025Kimi K2: Open Agentic Intelligencecited by 523
  2. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  3. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  4. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  5. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  6. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  7. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  8. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  9. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 158
  10. 2025xbench: Tracking Agents Productivity Scaling with Profession-Aligned Real-World Evaluationscited by 118
  11. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  12. 2025MiMo-Audio: Audio Language Models are Few-Shot Learnerscited by 58
38 more
  1. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  2. 2025lmgame-Bench: How Good are LLMs at Playing Games?cited by 54
  3. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 50
  4. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  5. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  6. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  7. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 41
  8. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  9. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  10. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  11. 2025PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoningcited by 29
  12. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 29
  13. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  14. 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoningcited by 28
  15. 2026STEP3-VL-10B Technical Reportcited by 28
  16. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  17. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  18. 2025Searching for Privacy Risks in LLM Agents via Simulationcited by 19
  19. 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Conditioncited by 19
  20. 2025Audio MultiChallenge: A Multi-Turn Evaluation of Spoken Dialogue Systems on Natural Human Interactioncited by 18
  21. 2026Reward Hacking in Rubric-Based Reinforcement Learningcited by 18
  22. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13
  23. 2026HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialoguecited by 12
  24. 2025TextQuests: How Good are LLMs at Text-Based Video Games?cited by 11
  25. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 11
  26. 2025EvaLearn: Quantifying the Learning Capability and Efficiency of LLMs via Sequential Problem Solvingcited by 10
  27. 2025MultiVerse: A Multi-Turn Conversation Benchmark for Evaluating Large Vision and Language Modelscited by 10
  28. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environmentscited by 9
  29. 2025One Battle After Another: Probing LLMs' Limits on Multi-Turn Instruction Following with a Benchmark Evolving Frameworkcited by 9
  30. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agentscited by 9
  31. 2026ERNIE 5.0 Technical Reportcited by 9
  32. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Modelscited by 9
  33. 2026Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifierscited by 9
  34. 2026From Question Answering to Task Completion: A Survey on Agent System and Harness Designcited by 9
  35. 2025TeachLM: Post-Training LLMs for Education Using Authentic Learning Datacited by 8
  36. 2025Simulating Viva Voce Examinations to Evaluate Clinical Reasoning in Large Language Modelscited by 8
  37. 2025Echoing: Identity Failures when LLM Agents Talk to Each Othercited by 8
  38. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluationcited by 8

50 most cited of 171 All 171 (JSON)