mlsanityData to 7 Oct 2026

The Impact of Positional Encoding on Length Generalization in Transformers

arXiv 2305.19466 ↗First posted 31 May 2023cs.CL cs.AI cs.LGcited by 319

Not on the map. Why?

Abstract (arXiv)

Length generalization, the ability to generalize from small training context sizes to larger ones, is a critical challenge in the development of Transformer-based language models. Positional encoding (PE) has been identified as a major factor influencing length generalization, but the exact impact of different PE schemes on extrapolation in downstream tasks remains unclear. In this paper, we conduct a systematic empirical study comparing the length generalization performance of decoder-only Transformers with five different position encoding approaches including Absolute Position Embedding (APE), T5's Relative PE, ALiBi, and Rotary, in addition to Transformers without positional encoding (NoPE). Our evaluation encompasses a battery of reasoning and mathematical tasks. Our findings reveal that the most commonly used positional encoding methods, such as ALiBi, Rotary, and APE, are not well suited for length generalization in downstream tasks. More importantly, NoPE outperforms other explicit positional encoding methods while requiring no additional computation. We theoretically demonstrate that NoPE can represent both absolute and relative PEs, but when trained with SGD, it mostly resembles T5's relative PE attention patterns. Finally, we find that scratchpad is not always helpful to solve length generalization and its format highly impacts the model's performance. Overall, our work suggests that explicit position embeddings are not essential for decoder-only Transformers to generalize well to longer sequences.

Key facts. Cited by 319 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

48 of 62 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2016Layer Normalizationcited by 5,579
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
36 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2018Self-Attention with Relative Position Representationscited by 1,112
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  5. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2022Galactica: A Large Language Model for Sciencecited by 736
  8. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  9. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  10. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  11. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  12. 2022Neural Networks and the Chomsky Hierarchycited by 214
  13. 2019Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernelcited by 211
  14. 2022Exploring Length Generalization in Large Language Modelscited by 204
  15. 2021Thinking Like Transformerscited by 182
  16. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  17. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  18. 2020From Local Structures to Size Generalization in Graph Neural Networkscited by 119
  19. 2015Neural GPUs Learn Algorithmscited by 118
  20. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  21. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  22. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  23. 2019Language Modeling with Deep Transformerscited by 96
  24. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  25. 2020Minimum Width for Universal Approximationcited by 92
  26. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  27. 2021Making Transformers Solve Compositional Taskscited by 72
  28. 2017DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understandingcited by 71
  29. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  30. 2020Measuring Systematic Generalization in Neural Proof Generation with Transformerscited by 43
  31. 2022Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysiscited by 39
  32. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  33. 2020Positional Artefacts Propagate Through Masked Language Model Embeddingscited by 34
  34. 2022The Curious Case of Absolute Position Embeddingscited by 17
  35. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  36. 2019Assessing the Ability of Self-Attention Networks to Learn Word Ordercited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 93; 2025: 103; 2026: 98.

319 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 331
  5. 2023Challenges and Applications of Large Language Modelscited by 251
  6. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  7. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  8. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  9. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  10. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  11. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  12. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
38 more
  1. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  2. 2025Cameras as Relative Positional Encodingcited by 94
  3. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  4. 2024Transformers need glasses! Information over-squashing in language taskscited by 87
  5. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  6. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  7. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  8. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  9. 2024Training-Free Long-Context Scaling of Large Language Modelscited by 61
  10. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  11. 2025On the Emergence of Position Bias in Transformerscited by 57
  12. 2023Positional Description Matters for Transformers Arithmeticcited by 56
  13. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  14. 2024Base of RoPE Bounds Context Lengthcited by 49
  15. 2025A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformerscited by 49
  16. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  17. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  18. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  19. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  20. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 41
  21. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  22. 2023In-Context Learning for Text Classification with Many Labelscited by 35
  23. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  24. 2023Quantum circuit synthesis with diffusion modelscited by 33
  25. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuningcited by 33
  26. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  27. 2024Mission: Impossible Language Modelscited by 31
  28. 2025Understanding Transformer from the Perspective of Associative Memorycited by 31
  29. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  30. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  31. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  32. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  33. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  34. 2024InfiniPot: Infinite Context Processing on Memory-Constrained LLMscited by 27
  35. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  36. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  37. 2025What are you sinking? A geometric approach on attention sinkcited by 26
  38. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25

50 most cited of 319 All 319 (JSON)