mlsanityData to 7 Oct 2026

NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks

arXiv 2204.05660 ↗First posted 12 Apr 2022cs.CL cs.AI cs.LGcited by 110

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Given the ubiquitous nature of numbers in text, reasoning with numbers to perform simple calculations is an important skill of AI systems. While many datasets and models have been developed to this end, state-of-the-art AI systems are brittle; failing to perform the underlying mathematical reasoning when they appear in a slightly different scenario. Drawing inspiration from GLUE that was proposed in the context of natural language understanding, we propose NumGLUE, a multi-task benchmark that evaluates the performance of AI systems on eight different tasks, that at their core require simple arithmetic understanding. We show that this benchmark is far from being solved with neural models including state-of-the-art large-scale language models performing significantly worse than humans (lower by 46.4%). Further, NumGLUE promotes sharing knowledge across tasks, especially those with limited training data as evidenced by the superior performance (average gain of 3.4% on each task) when a model is jointly trained on all the tasks as opposed to task-specific modeling. Finally, we hope that NumGLUE will encourage systems that perform robust and general arithmetic reasoning within language, a first step towards being able to perform more complex mathematical reasoning.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 40 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

28 of 40 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  9. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  10. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  11. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  12. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
16 more
  1. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  2. 2016Solving General Arithmetic Word Problemscited by 353
  3. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  4. 2021A Diverse Corpus for Evaluating and Developing English Math Word Problem Solverscited by 314
  5. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  6. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  7. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  8. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  9. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  10. 2018QuaRel: A Dataset and Models for Answering Questions about Qualitative Relationshipscited by 46
  11. 2019EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inferencecited by 39
  12. 2017Mapping to Declarative Knowledge for Word Problem Solvingcited by 36
  13. 2016Unit Dependency Graph and its Application to Arithmetic Word Problem Solvingcited by 25
  14. 2020DQI: Measuring Data Quality in NLPcited by 23
  15. 2020Our Evaluation Metric Needs an Update to Encourage Generalizationcited by 12
  16. 2019What's Missing: A Knowledge Gap Guided Approach for Multi-hop Question Answeringcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 22; 2024: 39; 2025: 18; 2026: 16.

110 papers.

Sort:

By area:

  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  3. 2023Augmented Language Models: a Surveycited by 398
  4. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  5. 2022FOLIO: Natural Language Reasoning with First-Order Logiccited by 233
  6. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  7. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  8. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  9. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  10. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  11. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 155
  12. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
38 more
  1. 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
  2. 2023Document Understanding Dataset and Evaluation (DUDE)cited by 132
  3. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  4. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  5. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  6. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  7. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 85
  8. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  9. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimizationcited by 68
  10. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Modelscited by 67
  11. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 59
  12. 2024Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuningcited by 59
  13. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  14. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  15. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  16. 2024SciInstruct: a Self-Reflective Instruction Annotated Dataset for Training Scientific Language Modelscited by 38
  17. 2024Pre-trained Large Language Models Use Fourier Features to Compute Additioncited by 38
  18. 2022Is a Question Decomposition Unit All We Need?cited by 37
  19. 2023Towards LogiGLUE: A Brief Survey and A Benchmark for Analyzing Logical Reasoning Capabilities of Language Modelscited by 35
  20. 2024GenQA: Generating Millions of Instructions from a Handful of Promptscited by 35
  21. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  22. 2023MultiTool-CoT: GPT-3 Can Use Multiple External Tools with Chain of Thought Promptingcited by 27
  23. 2024BAdam: A Memory Efficient Full Parameter Optimization Method for Large Language Modelscited by 27
  24. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  25. 2023Mathematical Language Models: A Surveycited by 24
  26. 2024Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlocited by 21
  27. 2025Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitionscited by 20
  28. 2024LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Modelscited by 19
  29. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19
  30. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  31. 2025Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methodscited by 18
  32. 2024ReGenesis: LLMs can Grow into Reasoning Generalists via Self-Improvementcited by 17
  33. 2022In-BoXBART: Get Instructions into Biomedical Multi-Task Learningcited by 16
  34. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  35. 2023Novice Learner and Expert Tutor: Evaluating Math Reasoning Abilities of Large Language Models with Misconceptionscited by 15
  36. 2024InfinityMATH: A Scalable Instruction Tuning Dataset in Programmatic Mathematical Reasoningcited by 15
  37. 2022Chaining Simultaneous Thoughts for Numerical Reasoningcited by 14
  38. 2022Lila: A Unified Benchmark for Mathematical Reasoningcited by 13

50 most cited of 110 All 110 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models
  2. 2026Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
  3. 2025Evaluating Mathematical Reasoning Across Large Language Models: A Fine-Grained Approach
  4. 2026Numeracy in Large Language Models: Fundamental Limitations and Paths to Improvement
  5. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  6. 2025Large Language Models in Numberland: A Quick Test of Their Numerical Reasoning Abilities
  7. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics
  8. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges