mlsanityData to 7 Oct 2026

Transformers Can Do Arithmetic with the Right Embeddings

arXiv 2405.17399 ↗First posted 27 May 2024cs.LG cs.AIcited by 77

Not on the map. Why?

Abstract (arXiv)

The poor performance of transformers on arithmetic tasks seems to stem in large part from their inability to keep track of the exact position of each digit inside of a large span of digits. We mend this problem by adding an embedding to each digit that encodes its position relative to the start of the number. In addition to the boost these embeddings provide on their own, we show that this fix enables architectural modifications such as input injection and recurrent layers to improve performance even further. With positions resolved, we can study the logical extrapolation ability of transformers. Can they solve arithmetic problems that are larger and more complex than those in their training data? We find that training on only 20 digit numbers with a single GPU for one day, we can reach state-of-the-art performance, achieving up to 99% accuracy on 100 digit addition problems. Finally, we show that these gains in numeracy also unlock improvements on other multi-step reasoning tasks including sorting and multiplication.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 43 of its 48 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

43 of 48 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2016Layer Normalizationcited by 5,579
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020GLU Variants Improve Transformercited by 1,724
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2021Scaling Vision Transformerscited by 993
31 more
  1. 2018Universal Transformerscited by 613
  2. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  4. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  5. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  6. 2023Looped Transformers as Programmable Computerscited by 224
  7. 2022Exploring Length Generalization in Large Language Modelscited by 204
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  9. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  10. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  11. 2019Adaptive Attention Span in Transformerscited by 148
  12. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 135
  13. 2023Teaching Arithmetic to Small Transformerscited by 121
  14. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  15. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  16. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  17. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  18. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  19. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  20. 2022Limitations of Language Models in Arithmetic and Symbolic Inductioncited by 71
  21. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  22. 2023Length Generalization in Arithmetic Transformerscited by 64
  23. 2022A Generalist Neural Algorithmic Learnercited by 58
  24. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  25. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  26. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  27. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
  28. 2024Simulation of Graph Algorithms with Looped Transformerscited by 29
  29. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  30. 2024Discrete Neural Algorithmic Reasoningcited by 8
  31. 2024Benchmarking ChatGPT on Algorithmic Reasoningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 32; 2026: 22.

77 papers.

Sort:

By area:

  1. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  2. 2025Any2AnyTryon: Leveraging Adaptive Position Embeddings for Versatile Virtual Clothing Taskscited by 62
  3. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  4. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 52
  5. 2024Procedural Knowledge in Pretraining Drives Reasoning in Large Language Modelscited by 41
  6. 2024Reconciling Kaplan and Chinchilla Scaling Lawscited by 36
  7. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  8. 2025Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformerscited by 28
  9. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  10. 2024How Numerical Precision Affects Arithmetical Reasoning Capabilities of LLMscited by 24
  11. 2025Encode, Think, Decode: Scaling test-time reasoning with recursive latent thoughtscited by 23
  12. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
38 more
  1. 2025FoNE: Precise Single-Token Number Embeddings via Fourier Featurescited by 21
  2. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  3. 2026Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoningcited by 17
  4. 2024LieRE: Lie Rotational Positional Encodingscited by 16
  5. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  6. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  7. 2025The Strawberry Problem: Emergence of Character-level Understanding in Tokenized Language Modelscited by 11
  8. 2025Revolution or Hype? Seeking the Limits of Large Models in Hardware Designcited by 9
  9. 2024Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythiacited by 8
  10. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  11. 2025Quantitative Bounds for Length Generalization in Transformerscited by 8
  12. 2024How language models extrapolate outside the training data: A case study in Textualized Gridworldcited by 7
  13. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMscited by 7
  14. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 7
  15. 2025Scaling Self-Supervised Representation Learning for Symbolic Piano Performancecited by 7
  16. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoningcited by 7
  17. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  18. 2025Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfactioncited by 6
  19. 2025Benford's Curse: Tracing Digit Bias to Numerical Hallucination in LLMscited by 6
  20. 2025Beyond Statistical Learning: Exact Learning Is Essential for General Intelligencecited by 6
  21. 2025Efficient numeracy in language models through single-token number embeddingscited by 6
  22. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Taskscited by 5
  23. 2024ENTP: Encoder-only Next Token Predictioncited by 5
  24. 2024Interleaving Text and Number Embeddings to Solve Mathemathics Problemscited by 5
  25. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  26. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 4
  27. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  28. 2025Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networkscited by 4
  29. 2024Towards a Science Exocortexcited by 3
  30. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 3
  31. 2024Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalizationcited by 2
  32. 2025GeoPE:A Unified Geometric Positional Embedding for Structured Tensorscited by 2
  33. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 2
  34. 2024Learning interpretable positional encodings in transformers depends on initializationcited by 1
  35. 2025Discovering Learning-Friendly Generation Orders for Sequential Computationcited by 1
  36. 2025CompoST: A Benchmark for Analyzing the Ability of LLMs To Compositionally Interpret Questions in a QALD Settingcited by 1
  37. 2026AI-rithmeticcited by 1
  38. 2026Training Transformers as a Universal Computercited by 1

50 most cited of 77 All 77 (JSON)