mlsanityData to 14 Oct 2026

Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPE

arXiv 2603.11611 ↗First posted 12 Mar 2026cs.LG cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Rotary Positional Embedding (RoPE) is widely used in transformers to encode relative positional information, yet the effect of applying RoPE to only a subset of hidden dimensions remains underexplored. This design choice can substantially reduce memory usage, especially at long context lengths. We show that partial RoPE achieves up to 10× memory savings over standard RoPE cache while maintaining comparable final loss. We systematically study its impact on training dynamics and convergence across architectures, model sizes, sequence lengths, and datasets. Our results reveal that: (1) applying RoPE to only a small fraction of dimensions (around 10%) matches the convergence of full RoPE; (2) these trends remain consistent across settings; and (3) NoPE (No Positional Encoding) can produce loss spikes, which are mitigated by minimal RoPE usage or QK-Norm application. Overall, our findings highlight partial RoPE as an effective trade-off between efficiency and stability.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 19 of its 37 references that are in the map. First posted on arXiv 12 Mar 2026.

Builds on

19 of 37 references.

References · 19

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2017Mixed Precision Trainingcited by 1,156
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  12. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
7 more
  1. 2020Query-Key Normalization for Transformerscited by 280
  2. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
  3. 2024Base of RoPE Bounds Context Lengthcited by 52
  4. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
  5. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 26
  6. 2025Hubble: a Model Suite to Advance the Study of LLM Memorizationcited by 17
  7. 2024When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Trainingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Disentangling the Expressivity of RoPEcited by 1
  2. 2026Does RoPE Prevent or Degrade Retrieval Heads? A Mechanistic Analysis Across Model Familiescited by 0
  3. 2026NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inferencecited by 0
  4. 2026Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrievalcited by 0