Frayed RoPE and Long Inputs: A Geometric Perspective
Not on the map. Why?
Abstract (arXiv)
Rotary Positional Embedding (RoPE) is a widely adopted technique for encoding position in language models, which, while effective, causes performance breakdown when input length exceeds training length. Prior analyses assert (rightly) that long inputs cause channels to rotate ``out of distribution,'' but it is not clear how extra rotation relates to or causes pathological behavior. Through empirical and theoretical analysis we advance a unified geometric understanding of attention behavior with RoPE. We find that attention induces tight clustering of separated key and query latent point clouds, allowing for creation of sink tokens: placeholders that allow attention heads to avoid token mixing when not required. RoPE applied to longer inputs damages this key/query cluster separation, producing pathological behavior by inhibiting sink token functionality. From this geometric perspective, we propose RoPE-ID (In Distribution), a straightforward modification that allows attention layers to generalize to longer inputs out of the box: apply RoPE with high frequency to a subset of channels. We demonstrate the effectiveness of RoPE-ID for extended inputs using 1B and 3B parameter Transformers on the LongBench and RULER information retrieval benchmarks.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
25 of 31 references.
References · 25
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
- 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
13 more
- 2024OLMo: Accelerating the Science of Language Modelscited by 599
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
- 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
- 2024Massive Activations in Large Language Modelscited by 277
- 2023Scaling Laws of RoPE-based Extrapolationcited by 124
- 2025Why do LLMs attend to the first token?cited by 122
- 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
- 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
- 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
- 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
- 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
- 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 25
- 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
Built on by
Citing papers per year: 2026: 3.
3 papers.