mlsanityData to 7 Oct 2026

LeRoPE: Learnable RoPE Frequencies Improve Language Modeling

arXiv 2607.10134 ↗First posted 11 Jul 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in RoPE typically follow a geometric sequence specified by a fixed base-frequency hyperparameter. Prior work has improved performance by either increasing this parameter to slow rotation or by applying RoPE to only a subset of QK dimensions. In this work we modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters. We validate Learned RoPE by training a ladder of language models from scratch, ranging from 52M to 2.5B parameters. We observe and analyze the emergence of a high-norm, positional LeRoPE band. LeRoPE consistently outperforms RoPE and partial RoPE across all scales, with RoPE requiring 3.4% more compute (FLOPs) to match LeRoPE at the largest scale.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 11 Jul 2026.

Builds on

24 of 30 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2018Self-Attention with Relative Position Representationscited by 1,112
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
12 more
  1. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  2. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  3. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  4. 2023Scaling Laws of RoPE-based Extrapolationcited by 124
  5. 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
  6. 2022Broken Neural Scaling Lawscited by 102
  7. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  8. 2024Base of RoPE Bounds Context Lengthcited by 49
  9. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 32
  10. 2025Communication-Efficient Language Model Training Scales Reliably and Robustly: Scaling Laws for DiLoCocited by 23
  11. 2024LieRE: Lie Rotational Positional Encodingscited by 16
  12. 2026Logarithmic-time Schedules for Scaling Language Models with Momentumcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Mesoscopic View of Transformer Weights Through Row and Column Scale Fieldscited by 0