mlsanityData to 7 Oct 2026

Randomized Positional Encodings Boost Length Generalization of Transformers

arXiv 2305.16843 ↗First posted 26 May 2023cs.LG cs.AI cs.CLcited by 114

Not on the map. Why?

Abstract (arXiv)

Transformers have impressive generalization capabilities on tasks with a fixed context length. However, they fail to generalize to sequences of arbitrary length, even for seemingly simple tasks such as duplicating a string. Moreover, simply training on longer sequences is inefficient due to the quadratic computation complexity of the global attention mechanism. In this work, we demonstrate that this failure mode is linked to positional encodings being out-of-distribution for longer sequences (even for relative encodings) and introduce a novel family of positional encodings that can overcome this problem. Concretely, our randomized positional encoding scheme simulates the positions of longer sequences and randomly selects an ordered subset to fit the sequence's length. Our large-scale empirical evaluation of 6000 models across 15 algorithmic reasoning tasks shows that our method allows Transformers to generalize to sequences of unseen length (increasing test accuracy by 12.0% on average).

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 29 of its 36 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

29 of 36 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022A Generalist Agentcited by 802
  10. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  11. 2018Universal Transformerscited by 613
  12. 2017Convolutional Sequence to Sequence Learningcited by 550
17 more
  1. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  2. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  3. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  4. 2022Neural Networks and the Chomsky Hierarchycited by 214
  5. 2020Random Features Strengthen Graph Neural Networkscited by 184
  6. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  7. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 110
  8. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
  9. 2021Making Transformers Solve Compositional Taskscited by 72
  10. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  11. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  12. 2022A Generalist Neural Algorithmic Learnercited by 58
  13. 2019Sequential Neural Networks as Automatacited by 54
  14. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  15. 2020How Can Self-Attention Networks Recognize Dyck-n Languages?cited by 51
  16. 2022Systematic Generalization and Emergent Structures in Transformers Trained on Structured Taskscited by 19
  17. 2020A Survey of Neural Networks and Formal Languagescited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 56; 2025: 29; 2026: 16.

114 papers.

Sort:

By area:

  1. 2023Language Modeling Is Compressioncited by 225
  2. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  3. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  4. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  5. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  6. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  7. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 84
  8. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  9. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
  10. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  11. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  12. 2023Positional Description Matters for Transformers Arithmeticcited by 56
38 more
  1. 2024LongEmbed: Extending Embedding Models for Long Context Retrievalcited by 47
  2. 2023CLEX: Continuous Length Extrapolation for Large Language Modelscited by 43
  3. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  4. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 33
  5. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  6. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  7. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  8. 2025Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challengescited by 30
  9. 2024The CLRS-Text Algorithmic Reasoning Language Benchmarkcited by 28
  10. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  11. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 25
  12. 2024ViTAR: Vision Transformer with Any Resolutioncited by 25
  13. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 25
  14. 2024Language Models Need Inductive Biases to Count Inductivelycited by 25
  15. 2024Universal Length Generalization with Turing Programscited by 25
  16. 2024DGR-MIL: Exploring Diverse Global Representation in Multiple Instance Learning for Whole Slide Image Classificationcited by 25
  17. 2024ALR2: A Retrieve-then-Reason Framework for Long-context Question Answeringcited by 25
  18. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  19. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  20. 2024Training Neural Networks as Recognizers of Formal Languagescited by 23
  21. 2024Long Context Alignment with Short Instructions and Synthesized Positionscited by 22
  22. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
  23. 2024Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannotcited by 21
  24. 2024TULIP: Token-length Upgraded CLIPcited by 18
  25. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  26. 2024An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encodingcited by 16
  27. 2024AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrievalcited by 16
  28. 2024Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generationcited by 15
  29. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  30. 2024FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Modelcited by 15
  31. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15
  32. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languagescited by 15
  33. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  34. 2023When Meta-Learning Meets Online and Continual Learning: A Surveycited by 12
  35. 2024VSTAR: Generative Temporal Nursing for Longer Dynamic Video Synthesiscited by 11
  36. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  37. 2023From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformerscited by 10
  38. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Taskscited by 10

50 most cited of 114 All 114 (JSON)