mlsanityData to 14 Oct 2026

Length Generalization of Causal Transformers without Position Encoding

arXiv 2404.12224 ↗First posted 18 Apr 2024cs.CLcited by 47

Not on the map. Why?

Abstract (arXiv)

Generalizing to longer sentences is important for recent Transformer-based language models. Besides algorithms manipulating explicit position features, the success of Transformers without position encodings (NoPE) provides a new way to overcome the challenge. In this paper, we study the length generalization property of NoPE. We find that although NoPE can extend to longer sequences than the commonly used explicit position encodings, it still has a limited context length. We identify a connection between the failure of NoPE's generalization and the distraction of attention distributions. We propose a parameter-efficient tuning for searching attention heads' best temperature hyper-parameters, which substantially expands NoPE's context size. Experiments on long sequence language modeling, the synthetic passkey retrieval task and real-world long context tasks show that NoPE can achieve competitive performances with state-of-the-art length generalization algorithms. The source code is publicly accessible

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). Builds on 20 of its 38 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

20 of 38 references.

References · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  9. 2023StarCoder: may the source be with you!cited by 799
  10. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  12. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
8 more
  1. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  2. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  3. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 136
  4. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  5. 2024Long Context Compression with Activation Beaconcited by 104
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2023A Survey on Long Text Modeling with Transformerscited by 47
  8. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 28; 2026: 10.

47 papers.

Sort:

By area:

  1. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  2. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  3. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  4. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  5. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  6. 2025Understanding Transformer from the Perspective of Associative Memorycited by 34
  7. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 31
  8. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 26
  9. 2025SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modelingcited by 19
  10. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18
  11. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 18
  12. 2024ReAttention: Training-Free Infinite Context with Finite Attention Scopecited by 13
35 more
  1. 2025Solving Empirical Bayes via Transformerscited by 11
  2. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  3. 2024Efficient Length-Generalizable Attention via Causal Retrieval for Long-Context Language Modelingcited by 9
  4. 2025Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Accesscited by 9
  5. 2025Position Bias Mitigates Position Bias:Mitigate Position Bias Through Inter-Position Knowledge Distillationcited by 9
  6. 2026RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provablycited by 8
  7. 2025Whither symbols in the era of advanced neural networks?cited by 7
  8. 2025Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Modelscited by 6
  9. 2025Deconstructing Positional Information: From Attention Logits to Training Biasescited by 5
  10. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Spacecited by 5
  11. 2025DoPE: Denoising Rotary Position Embeddingcited by 5
  12. 2025Group Representational Position Encodingcited by 5
  13. 2025Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encodingcited by 4
  14. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  15. 2025The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architecturescited by 4
  16. 2025RePo: Language Models with Context Re-Positioningcited by 4
  17. 2024Mitigate Position Bias in Large Language Models via Scaling a Single Dimensioncited by 3
  18. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  19. 2025RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modelingcited by 3
  20. 2025A Circular Argument : Does RoPE need to be Equivariant for Vision?cited by 3
  21. 2025Context-aware Biases for Length Extrapolationcited by 2
  22. 2025Positional Encoding via Token-Aware Phase Attentioncited by 1
  23. 2025Head-wise Adaptive Rotary Positional Encoding for Fine-Grained Image Generationcited by 1
  24. 2026Screening Is Enoughcited by 1
  25. 2025DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structurescited by 0
  26. 2025Home-made Diffusion Model from Scratch to Hatchcited by 0
  27. 2025An empirical study on the limitation of Transformers in program trace generationcited by 0
  28. 2025How Smoothing is N-simplicial Attention?cited by 0
  29. 2026HopFormer: Sparse Graph Transformers with Explicit Receptive Field Controlcited by 0
  30. 2026Dual Triangle Attention: Effective Bidirectional Attention Without Positional Embeddingscited by 0
  31. 2026Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extensioncited by 0
  32. 2026Resilient Concurrent Causal Discovery for Topological Event Sequencescited by 0
  33. 2026Multi-Head Self Attention is a Parameter Identification Mechanismcited by 0
  34. 2026VestigeKV: The NoPE-MLA KV Cache Carries Its Own Sparse-Attention Signal in a Vestigial Branchcited by 0
  35. 2026Asymptotic Behavior of Transformerscited by 0