mlsanityData to 14 Oct 2026

MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactions

arXiv 2405.19444 ↗First posted 29 May 2024cs.AIcited by 26

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models (LLMs) have demonstrated impressive capabilities in mathematical problem solving, particularly in single turn question answering formats. However, real world scenarios often involve mathematical question answering that requires multi turn or interactive information exchanges, and the performance of LLMs on these tasks is still underexplored. This paper introduces MathChat, a comprehensive benchmark specifically designed to evaluate LLMs across a broader spectrum of mathematical tasks. These tasks are structured to assess the models' abilities in multiturn interactions and open ended generation. We evaluate the performance of various SOTA LLMs on the MathChat benchmark, and we observe that while these models excel in single turn question answering, they significantly underperform in more complex scenarios that require sustained reasoning and dialogue understanding. To address the above limitations of existing LLMs when faced with multiturn and open ended tasks, we develop MathChat sync, a synthetic dialogue based math dataset for LLM finetuning, focusing on improving models' interaction and instruction following capabilities in conversations. Experimental results emphasize the need for training LLMs with diverse, conversational instruction tuning datasets like MathChatsync. We believe this work outlines one promising direction for improving the multiturn mathematical reasoning abilities of LLMs, thus pushing forward the development of LLMs that are more adept at interactive mathematical problem solving and real world applications.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 69 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

25 of 69 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  2. 2024Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models4 Mar
  3. 2024OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset15 Feb
  4. 2024InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning9 Feb
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models5 Feb
  6. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  7. 2023MinT: Boosting Generalization in Mathematical Reasoning via Multi-View Fine-Tuning16 Jul

Other references · 18

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2022Galactica: A Large Language Model for Sciencecited by 744
  10. 2023Instruction Tuning with GPT-4cited by 686
  11. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  12. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
6 more
  1. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  2. 2022Teaching Small Language Models to Reasoncited by 284
  3. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  4. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 68
  5. 2024Mathemyths: Leveraging Large Language Models to Teach Mathematical Language through Child-AI Co-Creative Storytellingcited by 21
  6. 2023Learning by Analogy: Diverse Questions Generation in Math Word Problemcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 7.

26 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  2. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  3. 2025MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutorscited by 49
  4. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verificationcited by 40
  5. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  6. 2024Unleashing LLM Reasoning Capability via Scalable Question Synthesis from Scratchcited by 24
  7. 2025EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Frameworkcited by 23
  8. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 21
  9. 2025Cultural Learning-Based Culture Adaptation of Language Modelscited by 16
  10. 2024Learn Beyond The Answer: Training Language Models with Reflection for Mathematical Reasoningcited by 15
  11. 2024FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedbackcited by 11
  12. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMscited by 10
14 more
  1. 2025Higher Satisfaction, Lower Cost: A Technical Report on How LLMs Revolutionize Meituan's Intelligent Interaction Systemscited by 7
  2. 2025Mitigating Lost in Multi-turn Conversation via Curriculum RL with Verifiable Accuracy and Abstention Rewardscited by 6
  3. 2026On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agentscited by 6
  4. 2025Reducing Belief Deviation in Reinforcement Learning for Active Reasoningcited by 4
  5. 2026FeedEval: Pedagogically Aligned Evaluation of LLM-Generated Essay Feedbackcited by 4
  6. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 3
  7. 2026From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Benchcited by 2
  8. 2025One-Pass to Reason: Token Duplication and Block-Sparse Mask for Efficient Fine-Tuning on Multi-Turn Reasoningcited by 1
  9. 2025Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Taskscited by 1
  10. 2026Confidence Should Be Calibrated More Than One Turn Deepcited by 1
  11. 2025Exploring Communication Strategies for Collaborative LLM Agents in Mathematical Problem-Solvingcited by 0
  12. 2026UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilitiescited by 0
  13. 2026When Does Critique Improve AI-Assisted Theoretical Physics? SCALAR: Structured Critic--Actor Loop for Agentic Reasoningcited by 0
  14. 2026Hy-MultiTurn: A Six-Dimensional Benchmark for Deep Multi-Turn Dialogue Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline
  2. 2024LLMs Are Not Intelligent Thinkers: Introducing Mathematical Topic Tree Benchmark for Comprehensive Evaluation of LLMs
  3. 2024Benchmarking Large Language Models for Math Reasoning Tasks
  4. 2025Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems
  5. 2024MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
  6. 2026MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
  7. 2024Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models
  8. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models