Disentangling the Expressivity of RoPE
Not on the map. Why?
Abstract (arXiv)
Two accounts recur in explanations of the success of rotary position embeddings (RoPE). Expressivity studies associate periodic position information with modular predicates, whereas mechanistic and long-context studies emphasize positional anchors and local offsets. We formalize both accounts for fully uniform, finite-precision soft-attention transformers. We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates. Conventional RoPE is different: The rotations it computes never repeat. This yields a precision-dependent bounded simulation of fixed-offset look-back operators, rather than an all-length modular characterization. Controlled experiments match this separation: Constructed periodic schedules length-generalize on modular languages, while conventional RoPE behaves more like a bounded locality bias and can impair tasks requiring position-invariant access to distant context. Altogether, our findings shed light on RoPE transformers, bringing theoretical expressivity characterizations closer to models used in practice.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 12 Aug 2026.
Builds on
29 of 42 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2025Qwen3 Technical Reportcited by 8,820
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
- 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
- 2022Transformers Learn Shortcuts to Automatacited by 268
- 2025Why do LLMs attend to the first token?cited by 127
- 2023Scaling Laws of RoPE-based Extrapolationcited by 125
- 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 115
- 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 93
- 2023Exposing Attention Glitches with Flip-Flop Language Modelingcited by 75
17 more
- 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 66
- 2024Base of RoPE Bounds Context Lengthcited by 52
- 2023Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languagescited by 49
- 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 47
- 2024Counting Like Transformers: Compiling Temporal Counting Logic Into Softmax Transformerscited by 39
- 2024Transformers Can Represent n-gram Language Modelscited by 32
- 2024Resonance RoPE: Improving Context Length Generalization of Large Language Modelscited by 27
- 2025Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddingscited by 26
- 2025Characterizing the Expressivity of Fixed-Precision Transformer Language Modelscited by 14
- 2025Knee-Deep in C-RASP: A Transformer Depth Hierarchycited by 12
- 2024Can Transformers Learn n-gram Language Models?cited by 11
- 2025Unique Hard Attention: A Tale of Two Sidescited by 10
- 2026Why Are Linear RNNs More Parallelizable?cited by 8
- 2026RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provablycited by 8
- 2026Fractional Rotation, Full Potential? Investigating Performance and Convergence of Partial RoPEcited by 4
- 2026Characterizing the Expressivity of Local Attention in Transformerscited by 3
- 2026Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don'tcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.