mlsanityData to 7 Oct 2026

The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval

arXiv 2502.11276 ↗First posted 16 Feb 2025cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

The Rotary Position Embedding (RoPE) is widely used in the attention heads of many large language models (LLM). It rotates dimensions in the query and the key vectors by different angles according to their positions in the input sequence. For long context modeling, the range of positions may vary a lot, and thus RoPE rotates some dimensions by a great range of angles. We hypothesize that the wide range of rotation angles may prevent LLMs from utilizing those dimensions. To validate this hypothesis, we present a controlled experiment showing that applying RoPE causes low utility of certain dimensions. Our analyses on three LLMs also indicate that these dimensions do not help LLMs do long-context question answering.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 22 of its 36 references that are in the map. First posted on arXiv 16 Feb 2025.

Builds on

22 of 36 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Qwen Technical Reportcited by 3,273
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  12. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
10 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  4. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  5. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  7. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  8. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283
  9. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  10. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 2
  2. 2026Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Familiescited by 0
  3. 2026Higher-Dimensional Rotary Position Embeddingcited by 0