mlsanityData to 7 Oct 2026

On the Limitations and Capabilities of Position Embeddings for Length Generalization

arXiv 2510.04130 ↗First posted 5 Oct 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In Transformers, Position Embeddings (PEs) significantly influence Length Generalization (LG) performance, yet their fundamental role remains unclear. In this work, we investigate the limitations and capabilities of PEs in achieving LG. We theoretically analyze PEs in Position-Only Linear Attentions (POLAs), introducing Linear Representation Complexity (LRC) to characterize when PEs enable LG. Our analysis shows that PEs do not expand computational capabilities but structure learned computations across positions. Extending to practical Transformers, we propose Sequential Representation Complexity (SRC) and conjecture that LG is possible if and only if SRC remains invariant across scales. We support this hypothesis with empirical evidence in various reasoning tasks. To enhance LG, we introduce Scale Hint, allowing flexible instance scaling, and a Learning-Based Position Embedding framework that automatically learns positional relations. Our work provides theoretical insights and practical strategies for improving LG in Transformers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

23 of 30 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2018Self-Attention with Relative Position Representationscited by 1,112
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  7. 2021Deep learning: a statistical viewpointcited by 239
  8. 2022Exploring Length Generalization in Large Language Modelscited by 204
  9. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  10. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  11. 2024Transformers Can Achieve Length Generalization But Not Robustlycited by 82
  12. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 77
11 more
  1. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  2. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  3. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  4. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  5. 2023Length Generalization in Arithmetic Transformerscited by 64
  6. 2024A Formal Framework for Understanding Length Generalization in Transformerscited by 44
  7. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  8. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  9. 2024LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Modelscited by 23
  10. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 22
  11. 2024Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalizationcited by 2

Built on by

Not cited yet.