mlsanityData to 7 Oct 2026

How well do Large Language Models perform in Arithmetic tasks?

arXiv 2304.02015 ↗First posted 16 Mar 2023cs.CL cs.AIcited by 130

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Large language models have emerged abilities including chain-of-thought to answer math word problems step by step. Solving math word problems not only requires abilities to disassemble problems via chain-of-thought but also needs to calculate arithmetic expressions correctly for each step. To the best of our knowledge, there is no work to focus on evaluating the arithmetic ability of large language models. In this work, we propose an arithmetic dataset MATH 401 to test the latest large language models including GPT-4, ChatGPT, InstrctGPT, Galactica, and LLaMA with various arithmetic expressions and provide a detailed analysis of the ability of large language models. MATH 401 and evaluation codes are released at urlhttps://github.com/GanjinZero/math401-llm.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 36 references that are in the map. First posted on arXiv 16 Mar 2023.

Builds on

29 of 36 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Language Models are Multilingual Chain-of-Thought Reasoners6 Oct
  2. 2020Generative Language Modeling for Automated Theorem Proving7 Sep

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
15 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  9. 2023Mathematical Capabilities of ChatGPTcited by 263
  10. 2022Autoformalization with Large Language Modelscited by 252
  11. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  12. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  13. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  14. 2021Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoningcited by 17
  15. 2021Pretrained Language Models are Symbolic Mathematics Solvers too!cited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 39; 2024: 45; 2025: 31; 2026: 15.

130 papers.

Sort:

By area:

  1. 2023Qwen Technical Reportcited by 3,273
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  4. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  5. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  7. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  8. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  9. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  10. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  11. 2024Large Language Models for Education: A Survey and Outlookcited by 185
  12. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
38 more
  1. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  2. 2023Teaching Arithmetic to Small Transformerscited by 121
  3. 2023Large Language Models are Few-Shot Health Learnerscited by 115
  4. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  5. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  6. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectivescited by 71
  7. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  8. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  9. 2023Autonomous GIS: the next-generation AI-powered GIScited by 52
  10. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 51
  11. 2023Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoningcited by 49
  12. 2024ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipelinecited by 47
  13. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 47
  14. 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languagescited by 46
  15. 2023MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoningcited by 45
  16. 2024Can LLMs Understand Time Series Anomalies?cited by 41
  17. 2025Reasoning Language Models: A Blueprintcited by 41
  18. 2023OptiMUS: Optimization Modeling Using MIP Solvers and large language modelscited by 39
  19. 2024Evaluating Large Language Models on Time Series Feature Understanding: A Comprehensive Taxonomy and Benchmarkcited by 38
  20. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  21. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  22. 2023Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBenchcited by 28
  23. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  24. 2024A Survey of Useful LLM Evaluationcited by 28
  25. 2024AutoGeo: Automating Geometric Image Dataset Creation for Enhanced Geometry Understandingcited by 28
  26. 2023Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement?cited by 26
  27. 2023Mathematical Language Models: A Surveycited by 24
  28. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  29. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  30. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 23
  31. 2025Brains vs. Bytes: Evaluating LLM Proficiency in Olympiad Mathematicscited by 22
  32. 2024Deeper Insights Without Updates: The Power of In-Context Learning Over Fine-Tuningcited by 20
  33. 2026Large Language Model Reasoning Failurescited by 20
  34. 2023Dissecting Chain-of-Thought: Compositionality through In-Context Filtering and Learningcited by 19
  35. 2024Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Surveycited by 19
  36. 2025VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Modelscited by 19
  37. 2023Unveiling Theory of Mind in Large Language Models: A Parallel to Single Neurons in the Human Braincited by 18
  38. 2024Causal Evaluation of Language Modelscited by 18

50 most cited of 130 All 130 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction
  2. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmetic
  3. 2024Improving Arithmetic Reasoning Ability of Large Language Models through Relation Tuples, Verification and Dynamic Feedback
  4. 2026Large Language Models for Math Education in Low-Resource Languages: A Study in Sinhala and Tamil
  5. 2025Evaluation of LLMs for mathematical problem solving
  6. 2024Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange
  7. 2025Large Language Models and Mathematical Reasoning Failures
  8. 2024Investigating Symbolic Capabilities of Large Language Models