# Attention on the Sphere

arXiv: https://arxiv.org/abs/2505.11157 · HTML: https://mlsanity.com/paper/2505.11157/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Neural Operator: Learning Maps Between Function Spaces](https://mlsanity.com/paper/2108.08481/) (2021)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs](https://mlsanity.com/paper/2206.11990/) (2022)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [Poseidon: Efficient Foundation Models for PDEs](https://mlsanity.com/paper/2405.19101/) (2024)
- [DeepSphere: a graph-based spherical CNN](https://mlsanity.com/paper/2012.15000/) (2020)
- [Neural Operators with Localized Integral and Differential Kernels](https://mlsanity.com/paper/2402.16845/) (2024)
- [Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Networks](https://mlsanity.com/paper/2310.06743/) (2023)
- [Spin-Weighted Spherical CNNs](https://mlsanity.com/paper/2006.10731/) (2020)
- [Efficient Generalized Spherical CNNs](https://mlsanity.com/paper/2010.11661/) (2020)
- [Advancing Parsimonious Deep Learning Weather Prediction using the HEALPix Mesh](https://mlsanity.com/paper/2311.06253/) (2023)
- [SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space](https://mlsanity.com/paper/2204.02394/) (2022)
- [VN-Transformer: Rotation-Equivariant Attention for Vector Neurons](https://mlsanity.com/paper/2206.04176/) (2022)
- [Scalable and Equivariant Spherical CNNs by Discrete-Continuous (DISCO) Convolutions](https://mlsanity.com/paper/2209.13603/) (2022)
- [Scaling Spherical CNNs](https://mlsanity.com/paper/2306.05420/) (2023)
- [Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level](https://mlsanity.com/paper/2403.04690/) (2024)
- [SphereUFormer: A U-Shaped Transformer for Spherical 360 Perception](https://mlsanity.com/paper/2412.06968/) (2024)
- [Spherical Transformer](https://mlsanity.com/paper/2202.04942/) (2022)
- [SE(3) Equivariant Ray Embeddings for Implicit Multi-View Depth Estimation](https://mlsanity.com/paper/2411.07326/) (2024)

## Built on by (most cited first, 10 of 10)

- [SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction](https://mlsanity.com/paper/2602.22867/) (2026)
- [MARA: Continuous SE(3)-Equivariant Attention for Molecular Force Fields](https://mlsanity.com/paper/2602.02671/) (2026)
- [Riemann GeoResolver: A Non-Euclidean Attention Framework from Euclidean Resolver to Hyperbolic-Spherical Geometry](https://mlsanity.com/paper/2608.10416/) (2026)
- [HRTFformer: A Spatially-Aware Transformer for Individual HRTF Upsampling in Immersive Audio Rendering](https://mlsanity.com/paper/2510.01891/) (2025)
- [Chem-GMNet: A Sphere-Native Geometric Transformer for Molecular Property Prediction](https://mlsanity.com/paper/2605.13262/) (2026)
- [DoRF++: Spherical Representation Learning over Doppler Radiance Fields for Robust Wi-Fi Sensing](https://mlsanity.com/paper/2608.08381/) (2026)
- [Dandelion: A Spherical Flower for Neural Simulation of Planetary Dynamics](https://mlsanity.com/paper/2608.27521/) (2026)
- [Refinement Symmetry in Multimodal Transformers](https://mlsanity.com/paper/2609.32669/) (2026)
- [A library for differentiable signal processing and machine learning on the sphere](https://mlsanity.com/paper/2609.39737/) (2026)
- [SO(3)-RoPE for Spherical Transformers](https://mlsanity.com/paper/2610.06229/) (2026)

## Abstract (arXiv)

We introduce a generalized attention mechanism for spherical domains, enabling Transformer architectures to natively process data defined on the two-dimensional sphere - a critical need in fields such as atmospheric physics, cosmology, and robotics, where preserving spherical symmetries and topology is essential for physical accuracy. By integrating numerical quadrature weights into the attention mechanism, we obtain a geometrically faithful spherical attention that is approximately rotationally equivariant, providing strong inductive biases and leading to better performance than Cartesian approaches. To further enhance both scalability and model performance, we propose neighborhood attention on the sphere, which confines interactions to geodesic neighborhoods. This approach reduces computational complexity and introduces the additional inductive bias for locality, while retaining the symmetry properties of our method. We provide optimized CUDA kernels and memory-efficient implementations to ensure practical applicability. The method is validated on three diverse tasks: simulating shallow water equations on the rotating sphere, spherical image segmentation, and spherical depth estimation. Across all tasks, our spherical Transformers consistently outperform their planar counterparts, highlighting the advantage of geometric priors for learning on spherical domains.