mlsanityData to 14 Oct 2026

SO(3)-RoPE for Spherical Transformers

arXiv 2610.06229 ↗First posted 5 Oct 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Spherical data arise in many scientific applications. Often spherical transformers disregard the geometry of the underlying spherical domain, causing distortions and coordinate singularities near the poles. We introduce SO(3)-RoPE, a relative positional embedding that incorporates spherical geometry into transformer attention through unitary SO(3) representations. Our formulation is SO(3)-equivariant and compatible with FlashAttention, retaining efficiency of vanilla transformers. On shallow water dynamics prediction over a rotating sphere, our SO3ViT outperforms an S2Transformer baseline with lower errors and reduced runtime.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 54 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

41 of 54 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  7. 2016Group Equivariant Convolutional Networkscited by 1,165
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  10. 2019General E(2)-Equivariant Steerable CNNscited by 446
  11. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 406
  12. 20183D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Datacited by 354
29 more
  1. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 327
  2. 2016Steerable CNNscited by 305
  3. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 300
  4. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 267
  5. 2017Learning SO(3) Equivariant Representations with Spherical CNNscited by 248
  6. 2023WeatherBench 2: A benchmark for the next generation of data-driven global weather modelscited by 236
  7. 2023Spherical Fourier Neural Operators: Learning Stable Dynamics on the Spherecited by 233
  8. 2017SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloudcited by 232
  9. 2018A General Theory of Equivariant CNNs on Homogeneous Spacescited by 228
  10. 2018Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Networkcited by 171
  11. 2019WoodScape: A multi-task, multi-camera fisheye dataset for autonomous drivingcited by 144
  12. 2022PanoFormer: Panorama Transformer for Indoor 360 Depth Estimationcited by 95
  13. 2019Spherical CNNs on Unstructured Gridscited by 79
  14. 2023GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformerscited by 79
  15. 2020DeepSphere: a graph-based spherical CNNcited by 66
  16. 2019Orientation-aware Semantic Segmentation on Icosahedron Spherescited by 53
  17. 2023EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimationcited by 43
  18. 2018SpherePHD: Applying CNNs on a Spherical PolyHeDron Representation of 360 degree Imagescited by 41
  19. 2022Scalable and Equivariant Spherical CNNs by Discrete-Continuous (DISCO) Convolutionscited by 24
  20. 2023HEAL-SWIN: A Vision Transformer On The Spherecited by 20
  21. 2023PanoSwin: a Pano-style Swin Transformer for Panorama Understandingcited by 20
  22. 2024SphereUFormer: A U-Shaped Transformer for Spherical 360 Perceptioncited by 13
  23. 2023Algebraic Positional Encodingscited by 10
  24. 2025Attention on the Spherecited by 10
  25. 2025Platonic Transformers: A Solid Choice For Equivariancecited by 10
  26. 2022SphereDepth: Panorama Depth Estimation from Spherical Domaincited by 7
  27. 2026(Sparse) Attention to the Details: Preserving Spectral Fidelity in ML-based Weather Forecasting Modelscited by 5
  28. 2025Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Embeddingcited by 4
  29. 2025A Circular Argument : Does RoPE need to be Equivariant for Vision?cited by 3

Built on by

Not cited yet.