mlsanityData to 7 Oct 2026

Transformers Can Achieve Length Generalization But Not Robustly

arXiv 2402.09371 ↗First posted 14 Feb 2024cs.LG cs.AI cs.CLcited by 82

Not on the map. Why?

Abstract (arXiv)

Length generalization, defined as the ability to extrapolate from shorter training sequences to longer test ones, is a significant challenge for language models. This issue persists even with large-scale Transformers handling relatively straightforward tasks. In this paper, we test the Transformer's ability of length generalization using the task of addition of two integers. We show that the success of length generalization is intricately linked to the data format and the type of position encoding. Using the right combination of data format and position encodings, we show for the first time that standard Transformers can extrapolate to a sequence length that is 2.5x the input length. Nevertheless, unlike in-distribution generalization, length generalization remains fragile, significantly influenced by factors like random weight initialization and training data order, leading to large variances across different random seeds.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 44 of its 52 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

44 of 52 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020GLU Variants Improve Transformercited by 1,724
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
32 more
  1. 2018Self-Attention with Relative Position Representationscited by 1,112
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2022In-context Learning and Induction Headscited by 930
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  6. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  7. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  8. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  9. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  10. 2022Transformers Learn Shortcuts to Automatacited by 260
  11. 2022Autoformalization with Large Language Modelscited by 252
  12. 2022Neural Networks and the Chomsky Hierarchycited by 214
  13. 2022Exploring Length Generalization in Large Language Modelscited by 204
  14. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  15. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  16. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  17. 2020Understanding the Failure Modes of Out-of-Distribution Generalizationcited by 146
  18. 2023Teaching Arithmetic to Small Transformerscited by 121
  19. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  20. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  21. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  22. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  23. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  24. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  25. 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 74
  26. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  27. 2023Length Generalization in Arithmetic Transformerscited by 64
  28. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  29. 2019Location Attention for Extrapolation to Longer Sequencescited by 33
  30. 2021A Data-Centric Approach for Training Deep Neural Networks with Less Datacited by 28
  31. 2023Improving Length-Generalization in Transformers via Task Hintingcited by 12
  32. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 33; 2026: 23.

82 papers.

Sort:

By area:

  1. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  2. 2025Position: Graph Learning Will Lose Relevance Due To Poor Benchmarkscited by 52
  3. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  4. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  5. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  6. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  7. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  8. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  9. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  10. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 25
  11. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  12. 2024Universal Length Generalization with Turing Programscited by 25
38 more
  1. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
  2. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 21
  3. 2024Mixture of Parrots: Experts improve memorization more than reasoningcited by 20
  4. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  5. 2024Quantifying Generalization Complexity for Large Language Modelscited by 18
  6. 2025Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Modelscited by 14
  7. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13
  8. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?cited by 13
  9. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  10. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  11. 2025Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalizationcited by 12
  12. 2024VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesiscited by 11
  13. 2024Your Context Is Not an Array: Unveiling Random Access Limitations in Transformerscited by 11
  14. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  15. 2024Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objectscited by 9
  16. 2025The Role of Sparsity for Length Generalization in Transformerscited by 8
  17. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 8
  18. 2024Adaptive Prediction Ensemble: Improving Out-of-Distribution Generalization of Motion Forecastingcited by 7
  19. 2024ALTA: Compiler-Based Analysis of Transformerscited by 7
  20. 2025Solving Empirical Bayes via Transformerscited by 7
  21. 2025Beyond Single-Task: Robust Multi-Task Length Generalization for LLMscited by 7
  22. 2025Random Scaling of Emergent Capabilitiescited by 7
  23. 2025On Vanishing Variance in Transformer Length Generalizationcited by 6
  24. 2025Efficient numeracy in language models through single-token number embeddingscited by 6
  25. 2025How Does RL Post-training Induce Skill Composition? A Case Study on Countdowncited by 6
  26. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  27. 2025RoboSSM: Scalable In-context Imitation Learning via State-Space Modelscited by 5
  28. 2025Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoningcited by 5
  29. 2026Eliciting Numerical Predictive Distributions of LLMs Without Autoregressioncited by 5
  30. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  31. 2025Learning to Add, Multiply, and Execute Algorithmic Instructions Exactly with Neural Networkscited by 4
  32. 2026Seeing to Generalize: How Visual Data Corrects Binding Shortcutscited by 4
  33. 2026Length Generalization Bounds for Transformerscited by 4
  34. 2024Executing Arithmetic: Fine-Tuning Large Language Models as Turing Machinescited by 3
  35. 2024Provable Length Generalization in Sequence Prediction via Spectral Filteringcited by 3
  36. 2025The Imitation Game: Turing Machine Imitator is Length Generalizable Reasonercited by 3
  37. 2025End-to-End Analysis of Charge Stability Diagrams with Transformerscited by 3
  38. 2025To Infinity and Beyond: Tool-Use Unlocks Length Generalization in State Space Modelscited by 3

50 most cited of 82 All 82 (JSON)