mlsanityData to 14 Oct 2026

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

arXiv 2605.15514 ↗First posted 15 May 2026cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models. Our theoretical analysis abstracts away from the specific content of the context and depends only on its length. We prove that as context length increases, RoPE-based attention becomes unpredictable and loses two properties that are central to its effectiveness. First, it loses its locality bias: RoPE is no more likely to favor nearer positions than substantially farther ones. Second, it loses consistency in token relevance: a key vector that receives a higher attention score than an alternative at one position may receive a lower score at another. In both cases, the probability of failure approaches 0.5, no better than random guessing. We further prove that the attention score can remain unchanged when a key token is moved to a different position, or even replaced by a different token, indicating a failure to distinguish positions or tokens. Adjusting the RoPE base trades off distinguishing positions against distinguishing tokens but cannot preserve both at the same time. Increasing the RoPE base hyperparameter, a common practice in today's long-context models, helps distinguish different tokens, but inevitably sacrifices the ability to distinguish positions. Our empirical analysis shows that multi-head, multi-layer architectures are insufficient to overcome these limitations. Our findings suggest that fundamentally new mechanisms for encoding position and token order may be needed in future Transformer long-context language models.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

30 of 42 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  10. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
18 more
  1. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  2. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  3. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  4. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  5. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  6. 2025Context Length Alone Hurts LLM Performance Despite Perfect Retrievalcited by 127
  7. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  8. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  9. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  10. 2024Base of RoPE Bounds Context Lengthcited by 52
  11. 2024Length Generalization of Causal Transformers without Position Encodingcited by 47
  12. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  13. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  14. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 26
  15. 2024Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspectivecited by 15
  16. 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
  17. 2026Rotary Positional Embeddings as Phase Modulation: Theoretical Bounds on the RoPE Base for Long-Context Transformerscited by 7
  18. 2025Rotary Offset Features in Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Disentangling the Expressivity of RoPEcited by 1
  2. 2026Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Familiescited by 0
  3. 2026When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodingscited by 0
  4. 2026Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrievalcited by 0
  5. 2026Scaling Parameter and Context in Attention: Native Sparse Attention from Mixture-of-Headcited by 0
  6. 2026When Context Changes: Understanding Update Failures in LLMscited by 0
  7. 2026RoPE at the End of Its Rope? Theory, Diagnosis, and Mitigation of Long-Context Failurescited by 0
  8. 2026WavePrune: One period is often enough for RoPEcited by 0