mlsanityData to 7 Oct 2026

MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation

arXiv 2312.17080 ↗First posted 28 Dec 2023cs.CLcited by 57

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this work, we introduce a novel evaluation paradigm for Large Language Models (LLMs) that compels them to transition from a traditional question-answering role, akin to a student, to a solution-scoring role, akin to a teacher. This paradigm, focusing on "reasoning about reasoning," hence termed meta-reasoning, shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation that effectively distinguishes between the cognitive capabilities of different models. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark. Our extensive analysis includes several state-of-the-art models from both open-source and commercial domains, uncovering fundamental deficiencies in their training and evaluation methodologies. Notably, while models like Deepseek-v2 and Claude3-Sonnet closely competed with GPT-4 in GSM8K, their performance disparities expanded dramatically in MR-GSM8K, with differences widening to over 20 absolute points, underscoring the significant challenge posed by our meta-reasoning approach.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 28 of its 36 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

28 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samples8 Nov
  2. 2023GPT-4 Can't Reason21 Jul
  3. 2022Language Models of Code are Few-Shot Commonsense Learners13 Oct

Other references · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
13 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  4. 2023Textbooks Are All You Needcited by 561
  5. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  6. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  7. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  8. 2020Generative Language Modeling for Automated Theorem Provingcited by 379
  9. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  10. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  11. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  12. 2023Three Questions Concerning the Use of Large Language Models to Facilitate Mathematics Learningcited by 9
  13. 2023VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistencycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 29; 2026: 20.

57 papers.

Sort:

By area:

  1. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 140
  2. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  3. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  4. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  5. 2024MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMscited by 30
  6. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  7. 2024Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Criticcited by 27
  8. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  9. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  10. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  11. 2025VERIFY: A Benchmark of Visual Explanation and Reasoning for Investigating Multimodal Reasoning Fidelitycited by 23
  12. 2024Evaluating LLMs' Mathematical and Coding Competency through Ontology-guided Interventionscited by 20
38 more
  1. 2024The Role of Deductive and Inductive Reasoning in Large Language Modelscited by 20
  2. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  3. 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Mathcited by 19
  4. 2025DeepCritic: Deliberate Critique with Large Language Modelscited by 17
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  6. 2025Scaling Evaluation-time Compute with Reasoning Models as Evaluatorscited by 13
  7. 2025MPBench: A Comprehensive Multimodal Reasoning Benchmark for Process Errors Identificationcited by 12
  8. 2025Error Classification of Large Language Models on Math Word Problems: A Dynamically Adaptive Frameworkcited by 10
  9. 2025Instruct-of-Reflection: Enhancing Large Language Models Iterative Reflection Capabilities via Dynamic-Meta Instructioncited by 10
  10. 2025Large Language Models Have Intrinsic Meta-Cognition, but Need a Good Lenscited by 10
  11. 2025Can You Trick the Grader? Adversarial Persuasion of LLM Judgescited by 9
  12. 2025Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmeticcited by 8
  13. 2026Learning Robust Reasoning through Guided Adversarial Self-Playcited by 8
  14. 2025LLMs cannot spot math errors, even when allowed to peek into the solutioncited by 7
  15. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  16. 2024Meta-Reasoning Improves Tool Use in Large Language Modelscited by 6
  17. 2025Beyond Correctness: Exposing LLM-generated Logical Flaws in Reasoning via Multi-step Automated Theorem Provingcited by 6
  18. 2025ProJudge: A Multi-Modal Multi-Discipline Benchmark and Instruction-Tuning Dataset for MLLM-based Process Judgescited by 5
  19. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  20. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  21. 2025ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answeringcited by 3
  22. 2026SimMerge: Learning to Select Merge Operators from Similarity Signalscited by 3
  23. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Taskscited by 3
  24. 2025Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluationcited by 2
  25. 2026Guided Collaboration in Heterogeneous LLM-Based Multi-Agent Systems via Entropy-Based Understanding Assessment and Experience Retrievalcited by 2
  26. 2026MedPRMBench: A Fine-grained Benchmark for Process Reward Models in Medical Reasoningcited by 2
  27. 2026GSM-SEM: Benchmark and Framework for Generating Semantically Variant Augmentationscited by 2
  28. 2025DeduCE: Deductive Consistency as a Framework to Evaluate LLM Reasoningcited by 1
  29. 2026From Solver to Tutor: Evaluating the Pedagogical Intelligence of LLMs with KMP-Benchcited by 1
  30. 2026How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environmentcited by 1
  31. 2026Decomposing the Delta: What Do Models Actually Learn from Preference Pairs?cited by 1
  32. 2026Metacognition in LLMs: Foundations, Progress, and Opportunitiescited by 1
  33. 2025SMART: Self-Generating and Self-Validating Multi-Dimensional Assessment for LLMs' Mathematical Problem Solvingcited by 0
  34. 2025MathBode: Measuring the Stability of LLM Reasoning using Frequency Responsecited by 0
  35. 2025PADME: Procedure Aware DynaMic Executioncited by 0
  36. 2025Accounting Reasoning in Large Language Models: Concepts, Evaluation, and Empirical Analysiscited by 0
  37. 2026Evaluating Accounting Reasoning Capabilities of Large Language Modelscited by 0
  38. 2026CoSA: Compressed Sensing-Based Adaptation of Large Language Modelscited by 0

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance
  2. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  3. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation
  4. 2025Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering
  5. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
  6. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  7. 2026Exploring the Meta-level Reasoning of Large Language Models via a Tool-based Multi-hop Tabular Question Answering Task
  8. 2024Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT