mlsanityData to 7 Oct 2026

Arithmetic with Language Models: from Memorization to Computation

arXiv 2308.01154 ↗First posted 2 Aug 2023cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation ineffective for novel data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing. Our findings support the hypothesis that the language model works as an Encoding-Regression-Decoding machine where the computation takes place in the value space once the input token representation is mapped to an appropriate internal representation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 22 of its 33 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

22 of 33 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  8. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  9. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  10. 2021Causal Abstractions of Neural Networkscited by 401
  11. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  12. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
10 more
  1. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  2. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  3. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  4. 2023Teaching Arithmetic to Small Transformerscited by 121
  5. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  6. 2023Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Taskscited by 84
  7. 2022Probing for the Usage of Grammatical Numbercited by 57
  8. 2021A Non-Linear Structural Probecited by 31
  9. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15
  10. 2022The activity-weight duality in feed forward neural networks: The geometric determinants of generalizationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 5; 2026: 4.

12 papers.

  1. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  2. 2024Language Models Encode Numbers Using Digit Representations in Base 10cited by 32
  3. 2025Probing for Arithmetic Errors in Language Modelscited by 17
  4. 2025Understanding In-context Learning of Addition via Activation Subspacescited by 8
  5. 2024OccamLLM: Fast and Exact Language Model Arithmetic in a Single Stepcited by 7
  6. 2025All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokenscited by 5
  7. 2025Language Models Do Not Embed Numbers Continuouslycited by 4
  8. 2026FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequencescited by 3
  9. 2025Deductive Logic in Language Models: Horizontal vs Vertical Reasoningcited by 1
  10. 2026A model of errors in transformerscited by 1
  11. 2026AI-rithmeticcited by 1
  12. 2026Bridging Graph Structure and Knowledge-Guided Editing for Interpretable Temporal Knowledge Graph Reasoningcited by 0