# Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks

arXiv: https://arxiv.org/abs/2406.02356 · HTML: https://mlsanity.com/paper/2406.02356/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning](https://mlsanity.com/paper/2002.06470/) (2020)
- [Investigating the Limitations of Transformers with Simple Arithmetic Tasks](https://mlsanity.com/paper/2102.13019/) (2021)
- [Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks](https://mlsanity.com/paper/2204.02892/) (2022)

## Built on by (most cited first, 24 of 24)

- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMs](https://mlsanity.com/paper/2502.19981/) (2025)
- [When Do Large Language Models Exhibit Unsolicited Deception?](https://mlsanity.com/paper/2504.00285/) (2025)
- [Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Steps](https://mlsanity.com/paper/2503.19326/) (2025)
- [In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Models](https://mlsanity.com/paper/2409.15454/) (2024)
- [Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls](https://mlsanity.com/paper/2510.00184/) (2025)
- [CASCADE: LLM-Powered JavaScript Deobfuscator at Google](https://mlsanity.com/paper/2507.17691/) (2025)
- [Learning Composable Chains-of-Thought](https://mlsanity.com/paper/2505.22635/) (2025)
- [Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2510.18196/) (2025)
- [A Minimum Description Length Approach to Regularization in Neural Networks](https://mlsanity.com/paper/2505.13398/) (2025)
- [OBsmith: LLM-Powered JavaScript Obfuscator Testing](https://mlsanity.com/paper/2510.10066/) (2025)
- [StreetMath: Study of LLMs' Approximation Behaviors](https://mlsanity.com/paper/2510.25776/) (2025)
- [AI-rithmetic](https://mlsanity.com/paper/2602.10416/) (2026)
- [PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations](https://mlsanity.com/paper/2502.07186/) (2025)
- [Grandes modelos de lenguaje: de la predicci'on de palabras a la comprensi'on?](https://mlsanity.com/paper/2502.18205/) (2025)
- [AGI Is Coming... Right After AI Learns to Play Wordle](https://mlsanity.com/paper/2504.15434/) (2025)
- [From General Reasoning to Domain Expertise: Uncovering the Limits of Generalization in Large Language Models](https://mlsanity.com/paper/2506.21580/) (2025)
- [Exploring NLP Benchmarks in an Extremely Low-Resource Setting](https://mlsanity.com/paper/2509.03962/) (2025)
- [Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Models](https://mlsanity.com/paper/2511.04053/) (2025)
- [From Prompt to Protocol: Fast Charging Batteries with Large Language Models](https://mlsanity.com/paper/2601.09626/) (2026)
- [On the Mirage of Long-Range Dependency, with an Application to Integer Multiplication](https://mlsanity.com/paper/2603.29069/) (2026)
- [When Tokenization is Secretly Output Supervision](https://mlsanity.com/paper/2609.01386/) (2026)
- [A Governance-Aware Large Language Model Orchestrated Agentic Digital Twin for Transmission System Operator Control Room Decision Support](https://mlsanity.com/paper/2609.22476/) (2026)
- [Which LLM to pick? Online Active Model Selection for Large Language Models](https://mlsanity.com/paper/2610.01592/) (2026)

## Abstract (arXiv)

The ability (and inability) of large language models (LLMs) to perform arithmetic tasks has been the subject of much theoretical and practical debate. We show that LLMs are frequently able to correctly and confidently predict the first digit of n-digit by m-digit multiplication tasks without using chain of thought reasoning, despite these tasks require compounding operations to solve. Simultaneously, LLMs in practice often fail to correctly or confidently predict the last digit of an n-digit by m-digit multiplication, a task equivalent to 1-digit by 1-digit multiplication which can be easily learned or memorized. We show that the latter task can be solved more robustly when the LLM is conditioned on all of the correct higher-order digits, which on average increases the confidence of the correct last digit on 5-digit by 5-digit multiplication tasks using Llama 2-13B by over 230% (0.13 to 0.43) and Mistral-7B by 150% (0.22 to 0.55).