RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways
Not on the map. Why?
Abstract (arXiv)
Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of its distance from the query. We propose RoVE, a parameter-free modification that makes values position-sensitive by rotating them simultaneously with keys, and show that it turns RoPE attention into attentive convolution. This new perspective unifies several independent formulations of the same operation across computer vision, robotics, and modern LLM architectures. Trained 124M and 354M GPT-2 models show consistent empirical gains over RoPE on few-shot in-context learning, out-of-distribution perplexity, and long-context retrieval, with the clearest improvements on tasks that require long-range aggregation.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 38 references that are in the map. First posted on arXiv 9 Jun 2026.
Builds on
33 of 38 references.
References · 33
- 2017Attention Is All You Needcited by 59,932
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2019Generating Long Sequences with Sparse Transformerscited by 1,523
- 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
21 more
- 2020SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networkscited by 602
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
- 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
- 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
- 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
- 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
- 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
- 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
- 2024Round and Round We Go! What makes Rotary Positional Encodings useful?cited by 111
- 2025Cameras as Relative Positional Encodingcited by 94
- 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
- 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 75
- 2020Attentive Group Equivariant Convolutional Networkscited by 62
- 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
- 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixerscited by 43
- 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
- 2024Beyond Canonicalization: How Tensorial Messages Improve Equivariant Message Passingcited by 13
- 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
- 2025Decoupling the "What" and "Where" With Polar Coordinate Positional Embeddingscited by 10
- 2026RayRoPE: Projective Ray Positional Encoding for Multi-view Attentioncited by 10
- 2026MrRoPE: Mixed-radix Rotary Position Embeddingcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.