mlsanityData to 14 Oct 2026

MrRoPE: Mixed-radix Rotary Position Embedding

arXiv 2601.22181 ↗First posted 28 Jan 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose MrRoPE (Mixed-radix RoPE), a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we introduce two training-free extensions, MrRoPE-Uni and MrRoPE-Pro, which leverage uniform and progressive radix conversion strategies, respectively, to achieve 'train short, test long' generalization. Without fine-tuning, MrRoPE-Pro sustains over 85% recall in the 128K-context Needle-in-a-Haystack test and achieves more than double YaRN's accuracy on Infinite-Bench retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pro effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 27 of its 32 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

27 of 32 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  9. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
15 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  5. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  6. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  8. 2022A Length-Extrapolatable Transformercited by 127
  9. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  10. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  11. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  12. 2024Base of RoPE Bounds Context Lengthcited by 52
  13. 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
  14. 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 27
  15. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathwayscited by 2
  2. 2026T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendationcited by 1
  3. 2026Anti-Periodic Positional Encoding: M"obius Boundary Conditions Make In-Context Retrieval Reliablecited by 0
  4. 2026WavePrune: One period is often enough for RoPEcited by 0