mlsanityData to 7 Oct 2026

Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Tasks

arXiv 2406.02356 ↗First posted 4 Jun 2024cs.LG cs.AI cs.CLcited by 24

Not on the map. Why?

Abstract (arXiv)

The ability (and inability) of large language models (LLMs) to perform arithmetic tasks has been the subject of much theoretical and practical debate. We show that LLMs are frequently able to correctly and confidently predict the first digit of n-digit by m-digit multiplication tasks without using chain of thought reasoning, despite these tasks require compounding operations to solve. Simultaneously, LLMs in practice often fail to correctly or confidently predict the last digit of an n-digit by m-digit multiplication, a task equivalent to 1-digit by 1-digit multiplication which can be easily learned or memorized. We show that the latter task can be solved more robustly when the LLM is conditioned on all of the correct higher-order digits, which on average increases the confidence of the correct last digit on 5-digit by 5-digit multiplication tasks using Llama 2-13B by over 230% (0.13 to 0.43) and Mistral-7B by 150% (0.22 to 0.55).

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 13 of its 30 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

13 of 30 references.

References · 13

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  9. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  10. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  11. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  12. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
1 more
  1. 2022Sub-Task Decomposition Enables Learning in Sequence to Sequence Taskscited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 16; 2026: 7.

24 papers.

Sort:

By area:

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2025The Lookahead Limitation: Why Multi-Operand Addition is Hard for LLMscited by 13
  3. 2025Process or Result? Manipulated Ending Tokens Can Mislead Reasoning LLMs to Ignore the Correct Reasoning Stepscited by 9
  4. 2025When Do Large Language Models Exhibit Unsolicited Deception?cited by 9
  5. 2024In-Context Learning May Not Elicit Trustworthy Reasoning: A-Not-B Errors in Pretrained Language Modelscited by 8
  6. 2025Why Can't Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfallscited by 6
  7. 2025CASCADE: LLM-Powered JavaScript Deobfuscator at Googlecited by 5
  8. 2025Learning Composable Chains-of-Thoughtcited by 3
  9. 2025Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judgecited by 3
  10. 2025A Minimum Description Length Approach to Regularization in Neural Networkscited by 2
  11. 2025OBsmith: LLM-Powered JavaScript Obfuscator Testingcited by 2
  12. 2025StreetMath: Study of LLMs' Approximation Behaviorscited by 2
12 more
  1. 2026AI-rithmeticcited by 1
  2. 2025PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relationscited by 0
  3. 2025Grandes modelos de lenguaje: de la predicci'on de palabras a la comprensi'on?cited by 0
  4. 2025AGI Is Coming... Right After AI Learns to Play Wordlecited by 0
  5. 2025From General Reasoning to Domain Expertise: Uncovering the Limits of Generalization in Large Language Modelscited by 0
  6. 2025Exploring NLP Benchmarks in an Extremely Low-Resource Settingcited by 0
  7. 2025Interpreting Multi-Attribute Confounding through Numerical Attributes in Large Language Modelscited by 0
  8. 2026From Prompt to Protocol: Fast Charging Batteries with Large Language Modelscited by 0
  9. 2026On the Mirage of Long-Range Dependency, with an Application to Integer Multiplicationcited by 0
  10. 2026When Tokenization is Secretly Output Supervisioncited by 0
  11. 2026A Governance-Aware Large Language Model Orchestrated Agentic Digital Twin for Transmission System Operator Control Room Decision Supportcited by 0
  12. 2026Which LLM to pick? Online Active Model Selection for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMs
  2. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics
  3. 2026The Effect of Scripts and Formats on LLM Numeracy
  4. 2024Unraveling Arithmetic in Large Language Models: The Role of Algebraic Structures
  5. 2025Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmetic
  6. 2022Limitations of Language Models in Arithmetic and Symbolic Induction
  7. 2025Large Language Models in Numberland: A Quick Test of Their Numerical Reasoning Abilities
  8. 2024Number Cookbook: Number Understanding of Language Models and How to Improve It