mlsanityData to 14 Oct 2026

Length Generalization in Arithmetic Transformers

arXiv 2306.15400 ↗First posted 27 Jun 2023cs.LGcited by 66

Not on the map. Why?

Abstract (arXiv)

We examine how transformers cope with two challenges: learning basic integer arithmetic, and generalizing to longer sequences than seen during training. We find that relative position embeddings enable length generalization for simple tasks, such as addition: models trained on 5-digit numbers can perform 15-digit sums. However, this method fails for multiplication, and we propose train set priming: adding a few (10 to 50) long sequences to the training set. We show that priming allows models trained on 5-digit × 3-digit multiplications to generalize to 35× 3 examples. We also show that models can be primed for different generalization lengths, and that the priming sample size scales as the logarithm of the training set size. Finally, we discuss potential applications of priming beyond arithmetic.

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

43 of 54 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
31 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  4. 2018Self-Attention with Relative Position Representationscited by 1,120
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  6. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  7. 2019Solving Rubik's Cube with a Robot Handcited by 734
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  10. 2020Generative Language Modeling for Automated Theorem Provingcited by 388
  11. 2019Deep Learning for Symbolic Mathematicscited by 270
  12. 2022Exploring Length Generalization in Large Language Modelscited by 206
  13. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  14. 2015Neural GPUs Learn Algorithmscited by 121
  15. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  16. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 112
  17. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 106
  18. 2015Inferring Algorithmic Patterns with Stack-Augmented Recurrent Netscited by 76
  19. 2020Improve Transformer Models with Better Relative Position Embeddingscited by 73
  20. 2018Neural Arithmetic Logic Unitscited by 70
  21. 2021Linear algebra with transformerscited by 63
  22. 2020Teaching Temporal Logics to Neural Networkscited by 51
  23. 2020The EOS Decision and Length Extrapolationcited by 46
  24. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  25. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  26. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  27. 2022SALSA: Attacking Lattice Cryptography with Transformerscited by 28
  28. 2022Simplifying Polylogarithms with Machine Learningcited by 15
  29. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  30. 2019Solving Math Word Problems with Double-Decoder Transformercited by 7
  31. 2021Solving Arithmetic Word Problems with Transformers and Preprocessing of Problem Textcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 24; 2025: 18; 2026: 14.

66 papers.

Sort:

By area:

  1. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  2. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
  3. 2023Amortizing intractable inference in large language modelscited by 97
  4. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 86
  5. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  6. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  7. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  8. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 47
  9. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  10. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 31
  11. 2024Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformerscited by 29
  12. 2024Universal Length Generalization with Turing Programscited by 25
38 more
  1. 2023Mathematical Language Models: A Surveycited by 24
  2. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 23
  3. 2024On Provable Length and Compositional Generalizationcited by 22
  4. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 21
  5. 2025Long-Context Generalization with Sparse Attentioncited by 21
  6. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 14
  7. 2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 14
  8. 2023Improving Length-Generalization in Transformers via Task Hintingcited by 13
  9. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  10. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  11. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Taskscited by 11
  12. 2025Graph neural networks extrapolate out-of-distribution for shortest pathscited by 11
  13. 2023Chain-of-Thought Reasoning is a Policy Improvement Operatorcited by 8
  14. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  15. 2024Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Modelscited by 7
  16. 2024Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Taskcited by 7
  17. 2025Random Scaling of Emergent Capabilitiescited by 7
  18. 2025Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilitiescited by 6
  19. 2025Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoningcited by 6
  20. 2025From Shortcut to Induction Head: How Data Diversity Shapes Algorithm Selection in Transformerscited by 6
  21. 2023Boolformer: Symbolic Regression of Logic Functions with Transformerscited by 5
  22. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Taskscited by 5
  23. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  24. 2024Making Hard Problems Easier with Custom Data Distributions and Loss Regularization: A Case Study in Modular Arithmeticcited by 4
  25. 2025Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networkscited by 4
  26. 2025Long-Short Alignment for Effective Long-Context Modeling in LLMscited by 4
  27. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 4
  28. 2026Seeing to Generalize: How Visual Data Corrects Binding Shortcutscited by 4
  29. 2026Length Generalization Bounds for Transformerscited by 4
  30. 2024Provable Length Generalization in Sequence Prediction via Spectral Filteringcited by 3
  31. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3
  32. 2026The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behaviorcited by 3
  33. 2026Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizercited by 3
  34. 2026The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thoughtcited by 3
  35. 2023It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Modelscited by 2
  36. 2024Machine learning for modular multiplicationcited by 2
  37. 2024Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalizationcited by 2
  38. 2024Three Pathways to Neurosymbolic Reinforcement Learning with Interpretable Model and Policy Networkscited by 1

50 most cited of 66 All 66 (JSON)