mlsanityData to 7 Oct 2026

How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs

arXiv 2410.13857 ↗First posted 17 Oct 2024cs.LG cs.AI cs.CLcited by 24

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Despite the remarkable success of Transformer-based large language models (LLMs) across various domains, understanding and enhancing their mathematical capabilities remains a significant challenge. In this paper, we conduct a rigorous theoretical analysis of LLMs' mathematical abilities, with a specific focus on their arithmetic performances. We identify numerical precision as a key factor that influences their effectiveness in arithmetical tasks. Our results show that Transformers operating with low numerical precision fail to address arithmetic tasks, such as iterated addition and integer multiplication, unless the model size grows super-polynomially with respect to the input length. In contrast, Transformers with standard numerical precision can efficiently handle these tasks with significantly smaller model sizes. We further support our theoretical findings through empirical experiments that explore the impact of varying numerical precision on arithmetic tasks, providing valuable insights for improving the mathematical reasoning capabilities of LLMs.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 88 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

30 of 88 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time25 May
  2. 2024CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities13 Jan
  3. 2023Learning From Mistakes Makes LLM Better Reasoner31 Oct
  4. 2023LPML: LLM-Prompting Markup Language for Mathematical Reasoning21 Sep

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
14 more
  1. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  2. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  3. 2019On the Turing Completeness of Modern Neural Network Architecturescited by 129
  4. 2023Solving Math Word Problems by Combining Language Models With Symbolic Solverscited by 124
  5. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  6. 2023MathChat: Converse to Tackle Challenging Math Problems with LLM Agentscited by 67
  7. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  8. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  9. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  10. 2023Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Modelscited by 35
  11. 2023An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)cited by 31
  12. 2020On the Ability and Limitations of Transformers to Recognize Formal Languagescited by 12
  13. 2023LLMs as Potential Brainstorming Partners for Math and Science Problemscited by 6
  14. 2023Rethinking Model-based, Policy-based, and Value-based Reinforcement Learning via the Lens of Representation Complexitycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 16; 2026: 6.

24 papers.

Sort:

By area:

  1. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 47
  2. 2025Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformerscited by 28
  3. 2025ReCode: Reinforcing Code Generation with Reasoning-Process Rewardscited by 25
  4. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Rangescited by 24
  5. 2025PDE-Controller: LLMs for Autoformalization and Reasoning of PDEscited by 23
  6. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  7. 2026Large Language Model Reasoning Failurescited by 20
  8. 2024Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokenscited by 18
  9. 2025MatterChat: A Multi-Modal LLM for Material Sciencecited by 18
  10. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  11. 2025Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoningcited by 7
  12. 2025From Sequence to Structure: Uncovering Substructure Reasoning in Transformerscited by 3
12 more
  1. 2025Trade-offs in Image Generation: How Do Different Dimensions Interact?cited by 3
  2. 2026TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Designcited by 3
  3. 2024Fine-Tuning Open-Weight Language Models to Deliver Cognitive Behavioral Therapy for Depression: A Feasibility Studycited by 1
  4. 2025Relative Overfitting and Accept-Reject Frameworkcited by 1
  5. 2025Analyzing the Power of Chain of Thought through Memorization Capabilitiescited by 1
  6. 2026A model of errors in transformerscited by 1
  7. 2026AI-rithmeticcited by 1
  8. 2025Univariate to Multivariate: LLMs as Zero-Shot Predictors for Time-Series Forecastingcited by 0
  9. 2025Basis Transformers for Multi-Task Tabular Regressioncited by 0
  10. 2025PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoningcited by 0
  11. 2026CIFQA: A Deterministic Tool-Grounded Multi-Agent LLM Framework for Financial Query Answeringcited by 0
  12. 2026When Tokenization is Secretly Output Supervisioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks
  2. 2026The Effect of Scripts and Formats on LLM Numeracy
  3. 2025A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs
  4. 2024Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures
  5. 2026Value-Aware Numerical Representations for Transformer Language Models
  6. 2025Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic
  7. 2026Testing LLM Arithmetic Reasoning Generalization with Automatic Numeric-Remapping Attacks
  8. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models