mlsanityData to 7 Oct 2026

Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap

arXiv 2402.19450 ↗First posted 29 Feb 2024cs.AI cs.CLcited by 79

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

We propose a framework for robust evaluation of reasoning capabilities of language models, using functional variants of benchmarks. Models that solve a reasoning test should exhibit no difference in performance over the static version of a problem compared to a snapshot of the functional variant. We have rewritten the relevant fragment of the MATH benchmark into its functional variant MATH(), with functionalization of other benchmarks to follow. When evaluating current state-of-the-art models over snapshots of MATH(), we find a reasoning gap -- the percentage difference between the static and functional accuracies. We find reasoning gaps from 58.35% to 80.31% among the state-of-the-art closed and open weights models that perform well on static benchmarks, with the caveat that the gaps are likely to be smaller with more sophisticated prompting strategies. Here we show that models which anecdotally have good reasoning performance over real-world tasks, have quantifiable lower gaps, motivating the open problem of building "gap 0" models. Code for evaluation and new evaluation datasets, three MATH() snapshots, are publicly available at https://github.com/consequentai/fneval/.

Key facts. Cited by 79 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 66 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

43 of 66 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models24 May

Other references · 42

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
30 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  8. 2023Teaching Large Language Models to Self-Debugcited by 963
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2023Large Language Modelscited by 774
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  14. 2018QuAC : Question Answering in Contextcited by 379
  15. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  16. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  17. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  18. 2023Planning with Large Language Models for Code Generationcited by 179
  19. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  20. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  21. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  22. 2023Chain of Code: Reasoning with a Language Model-Augmented Code Emulatorcited by 120
  23. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  24. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  25. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  26. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 88
  27. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 78
  28. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 49
  29. 2023Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscited by 45
  30. 2024Understanding Reasoning Ability of Language Models From the Perspective of Reasoning Paths Aggregationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 34; 2026: 7.

79 papers.

Sort:

By area:

  1. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 533
  2. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  3. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  4. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  5. 2024DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Modelscited by 143
  6. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  7. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  8. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  9. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  10. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  11. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  12. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 82
38 more
  1. 2025Towards Robust Mathematical Reasoningcited by 67
  2. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 60
  3. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 51
  4. 2025UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Modelscited by 45
  5. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
  6. 2025Reasoning Language Models: A Blueprintcited by 41
  7. 2025A Survey on Data Contamination for Large Language Modelscited by 33
  8. 2024Evaluating the Robustness of Analogical Reasoning in Large Language Modelscited by 31
  9. 2024VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbationcited by 29
  10. 2024Not All LLM Reasoners Are Created Equalcited by 29
  11. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  12. 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoningcited by 27
  13. 2024Accuracy is Not All You Needcited by 27
  14. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  15. 2024Can LLMs Solve longer Math Word Problems Better?cited by 22
  16. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  17. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  18. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  19. 2024Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarkscited by 17
  20. 2024UnSeenTimeQA: Time-Sensitive Question-Answering Beyond LLMs' Memorizationcited by 17
  21. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  22. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  23. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  24. 2025UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Modelscited by 13
  25. 2025RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluationcited by 11
  26. 2025EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solvingcited by 10
  27. 2024LLMs are not Zero-Shot Reasoners for Biomedical Information Extractioncited by 9
  28. 2025Putnam-AXIOM: A Functional and Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMscited by 9
  29. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Datacited by 8
  30. 2024Can Large Language Models Act as Symbolic Reasoners?cited by 8
  31. 2025VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarkscited by 8
  32. 2026Riemann-Bench: A Benchmark for Moonshot Mathematicscited by 8
  33. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoningcited by 7
  34. 2025Toward Automated Robustness Evaluation of Mathematical Reasoningcited by 6
  35. 2024Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Language Model Agentscited by 5
  36. 2024Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Studycited by 5
  37. 2025None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarkscited by 5
  38. 2025ASyMOB: Algebraic Symbolic Mathematical Operations Benchmarkcited by 5

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025ReliableMath: Benchmark of Reliable Mathematical Reasoning on Large Language Models
  2. 2025Catch Me If You Can: How Smaller Reasoning Models Pretend to Reason with Mathematical Fidelity
  3. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Models
  4. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  5. 2026More Accurate or More Efficient? Evaluating Locally Deployed Compact Open-Weight Language Models for Mathematical Reasoning
  6. 2025BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
  7. 2025MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning
  8. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges