# ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

arXiv: https://arxiv.org/abs/2506.03737 · HTML: https://mlsanity.com/paper/2506.03737/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action](https://mlsanity.com/paper/2312.17172/) (2023)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [Rethinking and Improving Relative Position Encoding for Vision Transformer](https://mlsanity.com/paper/2107.14222/) (2021)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)
- [WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences](https://mlsanity.com/paper/2306.07906/) (2023)
- [ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictions](https://mlsanity.com/paper/2403.07392/) (2024)
- [LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding](https://mlsanity.com/paper/2501.08282/) (2025)
- [Relative Positional Encoding for Transformers with Linear Complexity](https://mlsanity.com/paper/2105.08399/) (2021)
- [Distilling Semantic Priors from SAM to Efficient Image Restoration Models](https://mlsanity.com/paper/2403.16368/) (2024)
- [GIM: A Million-scale Benchmark for Generative Image Manipulation Detection and Localization](https://mlsanity.com/paper/2406.16531/) (2024)
- [Rethinking Pseudo-Label Guided Learning for Weakly Supervised Temporal Action Localization from the Perspective of Noise Correction](https://mlsanity.com/paper/2501.11124/) (2025)
- [IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning](https://mlsanity.com/paper/2502.02454/) (2025)
- [Translational Equivariance in Kernelizable Attention](https://mlsanity.com/paper/2102.07680/) (2021)

## Built on by (most cited first, 18 of 18)

- [A Circular Argument : Does RoPE need to be Equivariant for Vision?](https://mlsanity.com/paper/2511.08368/) (2025)
- [Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction](https://mlsanity.com/paper/2602.18996/) (2026)
- [Mitigating Coordinate Prediction Bias from Positional Encoding Failures](https://mlsanity.com/paper/2510.22102/) (2025)
- [Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE](https://mlsanity.com/paper/2511.11665/) (2025)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs](https://mlsanity.com/paper/2602.22716/) (2026)
- [SafeRoPE: Risk-specific Head-wise Embedding Rotation for Safe Generation in Rectified Flow Transformers](https://mlsanity.com/paper/2604.01826/) (2026)
- [MuViT: Multi-Resolution Vision Transformers for Learning Across Scales in Microscopy](https://mlsanity.com/paper/2602.24222/) (2026)
- [Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference](https://mlsanity.com/paper/2608.10288/) (2026)
- [Parabolic Position Encoding: Vision-Centric, Principled, Extrapolatable, General](https://mlsanity.com/paper/2602.01418/) (2026)
- [OPRO: Orthogonal Panel-Relative Operators for Panel-Aware In-Context Image Generation](https://mlsanity.com/paper/2603.27637/) (2026)
- [Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Cores](https://mlsanity.com/paper/2605.12491/) (2026)
- [Journey Operators for Structured Multi-Axis Composition](https://mlsanity.com/paper/2607.26775/) (2026)
- [When Local Variance Optimality Is Not Enough: RoPE-Aligned Q/K Rotations for Dynamic 4-Bit Quantisation](https://mlsanity.com/paper/2608.13365/) (2026)
- [Higher-Dimensional Rotary Position Embedding](https://mlsanity.com/paper/2608.29715/) (2026)
- [Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Models](https://mlsanity.com/paper/2609.30784/) (2026)
- [MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception](https://mlsanity.com/paper/2609.33804/) (2026)
- [WorldRoPE: Probabilistic World-Centric Rotary Embedding for Multi-View Attention](https://mlsanity.com/paper/2610.16337/) (2026)

## Abstract (arXiv)

The Transformer architecture has revolutionized various regions since it was proposed, and its effectiveness largely depends on the ability to encode positional information. Traditional position encoding methods exhibit significant limitations due to lack of robustness and flexibility of position. Therefore, Rotary Positional Encoding (RoPE) was proposed to alleviate these issues, which integrates positional information by rotating the embeddings in the attention mechanism. However, RoPE requires manually defined rotation matrices with limited transformation space, constraining the model's capacity. In this work, we propose ComRoPE, which generalizes RoPE by defining it in terms of trainable commuting angle matrices. Specifically, we demonstrate that pairwise commutativity of these matrices is essential for RoPE to achieve scalability and positional robustness. We formally define the RoPE Equation, which is an essential condition that ensures consistent performance with position offsets. Based on the theoretical analysis, we present two types of trainable commuting angle matrices as sufficient solutions to the RoPE equation, which significantly improve performance, surpassing the current state-of-the-art method by 1.6% at training resolution and 2.9% at higher resolution on the ImageNet-1K dataset. Furthermore, our framework shows versatility in generalizing to existing RoPE formulations and offering new insights for future positional encoding research. To ensure reproducibility, the source code and instructions are available at https://github.com/Longin-Yu/ComRoPE