# HEAL-SWIN: A Vision Transformer On The Sphere

arXiv: https://arxiv.org/abs/2307.07313 · HTML: https://mlsanity.com/paper/2307.07313/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 14 Jul 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation](https://mlsanity.com/paper/2105.05537/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Voxel Transformer for 3D Object Detection](https://mlsanity.com/paper/2109.02497/) (2021)
- [Embracing Single Stride 3D Object Detector with Sparse Transformer](https://mlsanity.com/paper/2112.06375/) (2021)
- [FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2208.05419/) (2022)
- [Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network](https://mlsanity.com/paper/1806.09231/) (2018)
- [Stratified Transformer for 3D Point Cloud Segmentation](https://mlsanity.com/paper/2203.14508/) (2022)
- [WoodScape: A multi-task, multi-camera fisheye dataset for autonomous driving](https://mlsanity.com/paper/1905.01489/) (2019)
- [Spherical Transformer for LiDAR-based 3D Recognition](https://mlsanity.com/paper/2303.12766/) (2023)
- [Spherical CNNs on Unstructured Grids](https://mlsanity.com/paper/1901.02039/) (2019)
- [DeepSphere: a graph-based spherical CNN](https://mlsanity.com/paper/2012.15000/) (2020)
- [Geometric Deep Learning and Equivariant Neural Networks](https://mlsanity.com/paper/2105.13926/) (2021)
- [DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications](https://mlsanity.com/paper/1810.12186/) (2018)
- [Orientation-aware Semantic Segmentation on Icosahedron Spheres](https://mlsanity.com/paper/1907.12849/) (2019)
- [Spin-Weighted Spherical CNNs](https://mlsanity.com/paper/2006.10731/) (2020)
- [SynWoodScape: Synthetic Surround-view Fisheye Camera Dataset for Autonomous Driving](https://mlsanity.com/paper/2203.05056/) (2022)
- [Efficient Generalized Spherical CNNs](https://mlsanity.com/paper/2010.11661/) (2020)
- [Equivariance versus Augmentation for Spherical Images](https://mlsanity.com/paper/2202.03990/) (2022)
- [Surface Networks via General Covers](https://mlsanity.com/paper/1812.10705/) (2018)
- [Equivariant Networks for Pixelized Spheres](https://mlsanity.com/paper/2106.06662/) (2021)

## Built on by (most cited first, 20 of 20)

- [AI Foundation Models for Weather and Climate: Applications, Design, and Implementation](https://mlsanity.com/paper/2309.10808/) (2023)
- [SphereUFormer: A U-Shaped Transformer for Spherical 360 Perception](https://mlsanity.com/paper/2412.06968/) (2024)
- [PEAR: Equal Area Weather Forecasting on the Sphere](https://mlsanity.com/paper/2505.17720/) (2025)
- [OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback](https://mlsanity.com/paper/2511.00510/) (2025)
- [FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras](https://mlsanity.com/paper/2509.13681/) (2025)
- [SO(3)-Equivariant Neural Networks for Learning from Scalar and Vector Fields on Spheres](https://mlsanity.com/paper/2503.09456/) (2025)
- [SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction](https://mlsanity.com/paper/2602.22867/) (2026)
- [Benchmarking Multi-View BEV Object Detection with Mixed Pinhole and Fisheye Cameras](https://mlsanity.com/paper/2603.27818/) (2026)
- [DarSwin-Unet: Distortion Aware Encoder-Decoder Architecture](https://mlsanity.com/paper/2407.17328/) (2024)
- [Boosting Omnidirectional Stereo Matching with a Pre-trained Depth Foundation Model](https://mlsanity.com/paper/2503.23502/) (2025)
- [Field-Space Attention for Structure-Preserving Earth System Transformers](https://mlsanity.com/paper/2512.20350/) (2025)
- [Efficiency Follows Global-Local Decoupling](https://mlsanity.com/paper/2603.19567/) (2026)
- [AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2608.29081/) (2026)
- [A Unified Geometric Field Theory Framework for Transformers: From Manifold Embeddings to Kernel Modulation](https://mlsanity.com/paper/2511.08243/) (2025)
- [Neural Operators for Biomedical Spherical Heterogeneity](https://mlsanity.com/paper/2601.03561/) (2026)
- [Distortion-Aware PETR for BEV Object Detection with Mixed Pinhole-Fisheye Cameras](https://mlsanity.com/paper/2606.08680/) (2026)
- [UniTriSplat: A Unified 3D Gaussian Splatting Framework with Uniform Spherical Rasterization for Universal Cameras](https://mlsanity.com/paper/2606.29794/) (2026)
- [QSV: Quat-Sphere-Vision for Coupled Quaternion Attention on Spherical Lattices](https://mlsanity.com/paper/2609.30592/) (2026)
- [S3N: A Spherical Spiral Scanning Network for Weather Forecasting](https://mlsanity.com/paper/2610.04338/) (2026)
- [SO(3)-RoPE for Spherical Transformers](https://mlsanity.com/paper/2610.06229/) (2026)

## Abstract (arXiv)

High-resolution wide-angle fisheye images are becoming more and more important for robotics applications such as autonomous driving. However, using ordinary convolutional neural networks or vision transformers on this data is problematic due to projection and distortion losses introduced when projecting to a rectangular grid on the plane. We introduce the HEAL-SWIN transformer, which combines the highly uniform Hierarchical Equal Area iso-Latitude Pixelation (HEALPix) grid used in astrophysics and cosmology with the Hierarchical Shifted-Window (SWIN) transformer to yield an efficient and flexible model capable of training on high-resolution, distortion-free spherical data. In HEAL-SWIN, the nested structure of the HEALPix grid is used to perform the patching and windowing operations of the SWIN transformer, enabling the network to process spherical representations with minimal computational overhead. We demonstrate the superior performance of our model on both synthetic and real automotive datasets, as well as a selection of other image datasets, for semantic segmentation, depth regression and classification tasks. Our code is publicly available at https://github.com/JanEGerken/HEAL-SWIN.