# How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs

arXiv: https://arxiv.org/abs/2410.13857 · HTML: https://mlsanity.com/paper/2410.13857/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 88 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Are Transformers universal approximators of sequence-to-sequence functions?](https://mlsanity.com/paper/1912.10077/) (2019)
- [Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models](https://mlsanity.com/paper/2408.00724/) (2024)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [On the Turing Completeness of Modern Neural Network Architectures](https://mlsanity.com/paper/1901.03429/) (2019)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [Learning From Mistakes Makes LLM Better Reasoner](https://mlsanity.com/paper/2310.20689/) (2023)
- [Transformers Can Do Arithmetic with the Right Embeddings](https://mlsanity.com/paper/2405.17399/) (2024)
- [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023)
- [RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval](https://mlsanity.com/paper/2402.18510/) (2024)
- [A Comprehensive Evaluation of Quantization Strategies for Large Language Models](https://mlsanity.com/paper/2402.16775/) (2024)
- [CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities](https://mlsanity.com/paper/2401.06961/) (2024)
- [How Does Quantization Affect Multilingual LLMs?](https://mlsanity.com/paper/2407.03211/) (2024)
- [MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time](https://mlsanity.com/paper/2405.16265/) (2024)
- [Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models](https://mlsanity.com/paper/2310.10449/) (2023)
- [An Independent Evaluation of ChatGPT on Mathematical Word Problems (MWP)](https://mlsanity.com/paper/2302.13814/) (2023)
- [LPML: LLM-Prompting Markup Language for Mathematical Reasoning](https://mlsanity.com/paper/2309.13078/) (2023)
- [On the Ability and Limitations of Transformers to Recognize Formal Languages](https://mlsanity.com/paper/2009.11264/) (2020)
- [LLMs as Potential Brainstorming Partners for Math and Science Problems](https://mlsanity.com/paper/2310.10677/) (2023)
- [Rethinking Model-based, Policy-based, and Value-based Reinforcement Learning via the Lens of Representation Complexity](https://mlsanity.com/paper/2312.17248/) (2023)

## Built on by (most cited first, 24 of 24)

- [Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference](https://mlsanity.com/paper/2506.09501/) (2025)
- [Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers](https://mlsanity.com/paper/2502.02393/) (2025)
- [ReCode: Reinforcing Code Generation with Reasoning-Process Rewards](https://mlsanity.com/paper/2508.05170/) (2025)
- [Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges](https://mlsanity.com/paper/2502.08680/) (2025)
- [PDE-Controller: LLMs for Autoformalization and Reasoning of PDEs](https://mlsanity.com/paper/2502.00963/) (2025)
- [FoNE: Precise Single-Token Number Embeddings via Fourier Features](https://mlsanity.com/paper/2502.09741/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens](https://mlsanity.com/paper/2411.17691/) (2024)
- [MatterChat: A Multi-Modal LLM for Material Science](https://mlsanity.com/paper/2502.13107/) (2025)
- [Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models](https://mlsanity.com/paper/2502.11075/) (2025)
- [Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning](https://mlsanity.com/paper/2505.11574/) (2025)
- [From Sequence to Structure: Uncovering Substructure Reasoning in Transformers](https://mlsanity.com/paper/2507.10435/) (2025)
- [Trade-offs in Image Generation: How Do Different Dimensions Interact?](https://mlsanity.com/paper/2507.22100/) (2025)
- [TIDE: Tuning-Integrated Dynamic Evolution for LLM-Based Automated Heuristic Design](https://mlsanity.com/paper/2601.21239/) (2026)
- [Fine-Tuning Open-Weight Language Models to Deliver Cognitive Behavioral Therapy for Depression: A Feasibility Study](https://mlsanity.com/paper/2412.00251/) (2024)
- [Relative Overfitting and Accept-Reject Framework](https://mlsanity.com/paper/2505.07783/) (2025)
- [Analyzing the Power of Chain of Thought through Memorization Capabilities](https://mlsanity.com/paper/2511.01190/) (2025)
- [A model of errors in transformers](https://mlsanity.com/paper/2601.14175/) (2026)
- [AI-rithmetic](https://mlsanity.com/paper/2602.10416/) (2026)
- [Univariate to Multivariate: LLMs as Zero-Shot Predictors for Time-Series Forecasting](https://mlsanity.com/paper/2506.02389/) (2025)
- [Basis Transformers for Multi-Task Tabular Regression](https://mlsanity.com/paper/2506.06926/) (2025)
- [PiERN: Token-Level Routing for Integrating High-Precision Computation and Reasoning](https://mlsanity.com/paper/2509.18169/) (2025)
- [CIFQA: A Deterministic Tool-Grounded Multi-Agent LLM Framework for Financial Query Answering](https://mlsanity.com/paper/2608.26114/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)

## Abstract (arXiv)

Despite the remarkable success of Transformer-based large language models (LLMs) across various domains, understanding and enhancing their mathematical capabilities remains a significant challenge. In this paper, we conduct a rigorous theoretical analysis of LLMs' mathematical abilities, with a specific focus on their arithmetic performances. We identify numerical precision as a key factor that influences their effectiveness in arithmetical tasks. Our results show that Transformers operating with low numerical precision fail to address arithmetic tasks, such as iterated addition and integer multiplication, unless the model size grows super-polynomially with respect to the input length. In contrast, Transformers with standard numerical precision can efficiently handle these tasks with significantly smaller model sizes. We further support our theoretical findings through empirical experiments that explore the impact of varying numerical precision on arithmetic tasks, providing valuable insights for improving the mathematical reasoning capabilities of LLMs.