mlsanityData to 7 Oct 2026

Round and Round We Go! What makes Rotary Positional Encodings useful?

arXiv 2410.06205 ↗First posted 8 Oct 2024cs.CL cs.LGcited by 111

Not on the map. Why?

Abstract (arXiv)

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 29 of its 40 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

29 of 40 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2023Vision Transformers Need Registerscited by 844
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
17 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  3. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  4. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  5. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  6. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  7. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  8. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  9. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  10. 2024Transformers need glasses! Information over-squashing in language taskscited by 87
  11. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  12. 2024How to think step-by-step: A mechanistic understanding of chain-of-thought reasoningcited by 53
  13. 2021Lower Perplexity is Not Always Human-Likecited by 52
  14. 2024Base of RoPE Bounds Context Lengthcited by 49
  15. 2024Length Generalization of Causal Transformers without Position Encodingcited by 46
  16. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  17. 2024Exploring Context Window of Large Language Models via Decomposed Positional Vectorscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 57; 2026: 51.

111 papers.

Sort:

By area:

  1. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
  2. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  3. 2026Gemma 4 Technical Reportcited by 187
  4. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  5. 2025Why do LLMs attend to the first token?cited by 122
  6. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  7. 2025On the Emergence of Position Bias in Transformerscited by 57
  8. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 47
  9. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  10. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  11. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 37
  12. 2025Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMscited by 27
38 more
  1. 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 23
  2. 2025HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Expertscited by 22
  3. 2025Hyperbolic Deep Learning for Foundation Models: A Surveycited by 20
  4. 2025Learning the RoPEs: Better 2D and 3D Position Encodings with STRINGcited by 19
  5. 2025Long-Context Generalization with Sparse Attentioncited by 18
  6. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  7. 2025Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddingscited by 16
  8. 2025Interpreting the Repeated Token Phenomenon in Large Language Modelscited by 15
  9. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  10. 2025Extrapolation by Association: Length Generalization Transfer in Transformerscited by 13
  11. 2025Panda: A pretrained forecast model for chaotic dynamicscited by 12
  12. 2026ReRoPE: Repurposing RoPE for Relative Camera Controlcited by 12
  13. 2026Addressable Memory for Video World Modelscited by 11
  14. 2025Hardware-Efficient Attention for Fast Decodingcited by 10
  15. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  16. 2025Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddingscited by 10
  17. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 10
  18. 2025AttentionPredictor: Temporal Patterns Matter for KV Cache Compressioncited by 7
  19. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 7
  20. 2025LLMs Process Lists With General Filter Headscited by 7
  21. 2025Synthesis of discrete-continuous quantum circuits with multimodal diffusion modelscited by 6
  22. 2025Decoupling Positional and Symbolic Attention Behavior in Transformerscited by 6
  23. 2026FASA: Frequency-aware Sparse Attentioncited by 6
  24. 2026RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provablycited by 6
  25. 2025Scale-invariant Attentioncited by 5
  26. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 5
  27. 2025Group Representational Position Encodingcited by 5
  28. 2025Deconstructing Positional Information: From Attention Logits to Training Biasescited by 4
  29. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Modelscited by 4
  30. 2026Why Attention Patterns Exist: A Unifying Temporal Perspective Analysiscited by 4
  31. 2025RePo: Language Models with Context Re-Positioningcited by 3
  32. 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 3
  33. 2026Decomposing Query-Key Feature Interactions Using Contrastive Covariancescited by 3
  34. 2026Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPEcited by 3
  35. 2026Frayed RoPE and Long Inputs: A Geometric Perspectivecited by 3
  36. 2026Generalizable Foundation Models for Calorimetry via Mixtures-of-Experts and Parameter Efficient Fine Tuningcited by 3
  37. 2026When Attention Closes: How LLMs Lose the Thread in Multi-Turn Interactioncited by 3
  38. 2026Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformerscited by 3

50 most cited of 111 All 111 (JSON)