# SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction

arXiv: https://arxiv.org/abs/2602.22867 · HTML: https://mlsanity.com/paper/2602.22867/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 61 references that are in the map.
- First posted on arXiv 26 Feb 2026.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Geometric deep learning: going beyond Euclidean data](https://mlsanity.com/paper/1611.08097/) (2016)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds](https://mlsanity.com/paper/1802.08219/) (2018)
- [SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks](https://mlsanity.com/paper/2006.10503/) (2020)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age](https://mlsanity.com/paper/1606.05830/) (2016)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019)
- [DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications](https://mlsanity.com/paper/1810.12186/) (2018)
- [Review on Panoramic Imaging and Its Applications in Scene Understanding](https://mlsanity.com/paper/2205.05570/) (2022)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [SphereSR: 360deg Image Super-Resolution with Arbitrary Projection via Continuous Spherical Image Representation](https://mlsanity.com/paper/2112.06536/) (2021)
- [Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion](https://mlsanity.com/paper/2403.16376/) (2024)
- [SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation](https://mlsanity.com/paper/2306.03403/) (2023)

## Built on by (most cited first, 2 of 2)

- [Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling](https://mlsanity.com/paper/2606.27745/) (2026)
- [PanoPed: Beyond Bounding Boxes for Sim-to-Real Panoramic Pedestrian Tracking](https://mlsanity.com/paper/2610.08826/) (2026)

## Abstract (arXiv)

Panoramic dense-prediction models, spanning semantic segmentation and depth estimation, are typically trained under a strict gravity-aligned assumption. Real-world captures, however, routinely violate it: handheld devices jitter and aerial platforms change attitude, so the camera is rarely upright. Under such 3D reorientation, standard spherical Transformers overfit global latitude cues and collapse. We introduce SO3UFormer, an architecture that learns intrinsic spherical features largely decoupled from the underlying coordinate frame, through three geometric components: (1) removing absolute latitude encoding, which breaks the dependence on the gravity axis; (2) quadrature-consistent spherical attention, which corrects for non-uniform sampling density; and (3) a gauge-aware relative positional bias built from local tangent-plane angles rather than global axes. A logit-space SO(3)-consistency regularizer, used only during training, further suppresses residual discretization effects. To benchmark robustness, we introduce Pose35, a variant of Stanford2D3D perturbed by random rotations within ± 35^circ, and evaluate under a full, arbitrary SO(3) stress test. There, the baseline SphereUFormer collapses from 67.53 mIoU on Pose35 to 25.26 under the full SO(3) test, whereas SO3UFormer reaches 72.03 on Pose35 and retains 70.67 under the same test. Similarly, on a second real-world dataset (Matterport3D) for segmentation and on panoramic depth estimation, SO3UFormer remains essentially rotation-invariant while the gravity-anchored baseline again loses most of its accuracy. Code and models are available at https://github.com/zhuqinfeng1999/SO3UFormer.