# SO(3)-RoPE for Spherical Transformers

arXiv: https://arxiv.org/abs/2610.06229 · HTML: https://mlsanity.com/paper/2610.06229/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 41 of its 54 references that are in the map.
- First posted on arXiv 5 Oct 2026.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges](https://mlsanity.com/paper/2104.13478/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data](https://mlsanity.com/paper/1807.02547/) (2018)
- [The Impact of Positional Encoding on Length Generalization in Transformers](https://mlsanity.com/paper/2305.19466/) (2023)
- [Steerable CNNs](https://mlsanity.com/paper/1612.08498/) (2016)
- [On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups](https://mlsanity.com/paper/1802.03690/) (2018)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)
- [A General Theory of Equivariant CNNs on Homogeneous Spaces](https://mlsanity.com/paper/1811.02017/) (2018)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)
- [WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving](https://mlsanity.com/paper/1905.01489/) (2019)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [Spherical CNNs on Unstructured Grids](https://mlsanity.com/paper/1901.02039/) (2019)
- [GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers](https://mlsanity.com/paper/2310.10375/) (2023)
- [DeepSphere: a graph-based spherical CNN](https://mlsanity.com/paper/2012.15000/) (2020)
- [Orientation-aware Semantic Segmentation on Icosahedron Spheres](https://mlsanity.com/paper/1907.12849/) (2019)
- [EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2304.07803/) (2023)
- [SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Images](https://mlsanity.com/paper/1811.08196/) (2018)

## Abstract (arXiv)

Spherical data arise in many scientific applications. Often spherical transformers disregard the geometry of the underlying spherical domain, causing distortions and coordinate singularities near the poles. We introduce SO(3)-RoPE, a relative positional embedding that incorporates spherical geometry into transformer attention through unitary SO(3) representations. Our formulation is SO(3)-equivariant and compatible with FlashAttention, retaining efficiency of vanilla transformers. On shallow water dynamics prediction over a rotating sphere, our SO3ViT outperforms an S2Transformer baseline with lower errors and reduced runtime.