mlsanityData to 7 Oct 2026

From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers

arXiv 2310.11984 ↗First posted 18 Oct 2023cs.LG cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the inherent capabilities of transformer models in learning arithmetic algorithms, such as addition and parity. Through experiments and attention analysis, we identify a number of crucial factors for achieving optimal length generalization. We show that transformer models are able to generalize to long lengths with the help of targeted attention biasing. In particular, our solution solves the Parity task, a well-known and theoretically proven failure mode for Transformers. We then introduce Attention Bias Calibration (ABC), a calibration stage that enables the model to automatically learn the proper attention biases, which we show to be connected to mechanisms in relative position encoding. We demonstrate that using ABC, the transformer model can achieve unprecedented near-perfect length generalization on certain arithmetic tasks. In addition, we show that ABC bears remarkable similarities to RPE and LoRA, which may indicate the potential for applications to more complex tasks.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

25 of 33 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2018Self-Attention with Relative Position Representationscited by 1,112
  5. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  8. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  9. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  10. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  11. 2022Neural Networks and the Chomsky Hierarchycited by 214
  12. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
13 more
  1. 2022Exploring Length Generalization in Large Language Modelscited by 204
  2. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  3. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  4. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  5. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  6. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  7. 2021Position Information in Transformers: An Overviewcited by 103
  8. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  9. 2020Memformer: A Memory-Augmented Transformer for Sequence Modelingcited by 63
  10. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  11. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  12. 2018Input Combination Strategies for Multi-Source Transformer Decodercited by 22
  13. 2017The cognitive roots of regularization in languagecited by 5

Built on by

Cited by year

Citing papers per year: 2024: 6; 2025: 4.

10 papers.

  1. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  2. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  3. 2024Tools Fail: Detecting Silent Errors in Faulty Toolscited by 23
  4. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
  5. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  6. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  7. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Taskscited by 5
  8. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3
  9. 2024Re-Tuning: Overcoming the Compositionality Limits of Large Language Models with Recursive Tuningcited by 2
  10. 2025Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolationcited by 0