mlsanityData to 14 Oct 2026

Selective Rotary Position Embedding

arXiv 2511.17388 ↗First posted 21 Nov 2025cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Position information is essential for language modeling. In softmax transformers, Rotary Position Embeddings (textitRoPE) encode positions through textitfixed-angle rotations, while in linear transformers, order is handled via input-dependent (selective) gating that decays past key-value associations. Selectivity has generally been shown to improve language-related tasks. Inspired by this, we introduce textitSelective RoPE, an textitinput-dependent rotary embedding mechanism, that generalizes textitRoPE, and enables rotation in textitarbitrary angles for both linear and softmax transformers. We show that softmax attention already performs a hidden form of these rotations on query-key pairs, uncovering an implicit positional structure. We further show that in state-space models and gated linear transformers, the real part manages forgetting while the imaginary part encodes positions through rotations. We validate our method by equipping gated transformers with textitSelective RoPE, demonstrating that its input-dependent rotations improve performance in language modeling and on difficult sequence tasks like copying, state tracking, and retrieval.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 56 of its 72 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

56 of 72 references.

References · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  11. 2020Rethinking Attention with Performerscited by 1,601
  12. 2018On the Spectral Bias of Neural Networkscited by 1,341
38 more
  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2018Self-Attention with Relative Position Representationscited by 1,120
  4. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  5. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  8. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 543
  10. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  12. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  13. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  14. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  15. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  16. 2021Random Feature Attentioncited by 354
  17. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  18. 2024xLSTM: Extended Long Short-Term Memorycited by 327
  19. 2020Query-Key Normalization for Transformerscited by 280
  20. 2022Transformers Learn Shortcuts to Automatacited by 268
  21. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  22. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  23. 2024Simple linear attention language models balance the recall-throughput tradeoffcited by 208
  24. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  25. 2024The Illusion of State in State-Space Modelscited by 187
  26. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  27. 2025RWKV-7 "Goose" with Expressive Dynamic State Evolutioncited by 141
  28. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  29. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  30. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 120
  31. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  32. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  33. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 86
  34. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  35. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 59
  36. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 53
  37. 2025In Search of Adam's Secret Saucecited by 40
  38. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 36

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 6
  2. 2025RePo: Language Models with Context Re-Positioningcited by 4
  3. 2026T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendationcited by 1
  4. 2026Beyond ZOH: Advanced Discretization Strategies for Vision Mambacited by 0
  5. 2026Why Do Accumulated Transformations Extrapolate?cited by 0
  6. 2026RoPE is Dead, Long Live RoPE: Towards Scalable Data-aware Positional Encodingscited by 0
  7. 2026CyFA: Linear Sequence Modeling with Relative-Time-Partitioned Memorycited by 0
  8. 2026Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrievalcited by 0
  9. 2026Understanding Mamba-3 Through Oscillatory Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VRoPE: Rotary Position Embedding for Video Large Language Models
  2. 2026ZetaGPT: A Reference Implementation of Positional--Encoding--Free State--Space--Attention Language Models
  3. 2025SeqPE: Transformer with Sequential Position Encoding
  4. 2026LinearARD: Linear-Memory Attention Distillation for RoPE Restoration
  5. 2025Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models
  6. 2025Revisiting Multimodal Positional Encoding in Vision-Language Models
  7. 2026Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers
  8. 2026Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization