How Data Shapes RoPE Frequency Usage: From Positional Scale Matching to Length Generalization
Not on the map. Why?
Abstract (arXiv)
Rotary Position Embeddings (RoPE) provide transformers with a fixed grid of positional frequencies, yet trained models use these frequencies highly non-uniformly. We study what determines this frequency usage and propose a data-centered explanation: RoPE frequencies are selected to match the relative-distance structure of the training data. Viewing each frequency as a positional lens, we formalize a field-resolution tradeoff and show that, for a data-induced dependency profile of width W, the optimal frequency scales as 1/W. This frequency-matching principle explains controlled observations on synthetic and text-based data, and suggests that the mid-low frequency bands observed in language models arise from the multi-scale dependency structure of natural language. We further connect frequency selection to position-interpolation-based length generalization: scaling frequencies down expands the effective field while reducing resolution. This helps when longer-context dependencies are approximate dilations of those seen during training, but can fail when relevant dependencies do not scale with context length. Empirically, we show that natural language exhibits approximate self-similarity across positional scales, explaining why test-time frequency scaling can support long-context generalization. Overall, our results identify a data-driven mechanism behind emergent RoPE frequency usage and show that long-context generalization depends on two forms of scale matching: between learned frequencies and training-time dependencies, and between frequency scaling and how those dependencies extend to longer contexts.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 8 Jul 2026.
Builds on
33 of 48 references.
References · 33
- 2017Attention Is All You Needcited by 60,361
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
21 more
- 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
- 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
- 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
- 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
- 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 118
- 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
- 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
- 2017Structured Attention Networkscited by 95
- 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 83
- 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
- 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 65
- 2025On the Emergence of Position Bias in Transformerscited by 59
- 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 48
- 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
- 2024Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalizationcited by 39
- 2024Serial Position Effects of Large Language Modelscited by 30
- 2025Rethinking Invariance in In-context Learningcited by 8
- 2024Fractal Patterns May Illuminate the Success of Next-Token Predictioncited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.