# SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation

arXiv: https://arxiv.org/abs/2306.03403 · HTML: https://mlsanity.com/paper/2306.03403/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 45 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer](https://mlsanity.com/paper/2108.01390/) (2021)
- [Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2105.15075/) (2021)
- [HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features](https://mlsanity.com/paper/2011.11498/) (2020)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [Spherical CNNs on Unstructured Grids](https://mlsanity.com/paper/1901.02039/) (2019)
- [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019)
- [Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation](https://mlsanity.com/paper/2203.01452/) (2022)
- [Spin-Weighted Spherical CNNs](https://mlsanity.com/paper/2006.10731/) (2020)
- [Capturing Omni-Range Context for Omnidirectional Segmentation](https://mlsanity.com/paper/2103.05687/) (2021)
- [Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2207.11860/) (2022)
- [Vision Transformer with Progressive Sampling](https://mlsanity.com/paper/2108.01684/) (2021)
- [DPT: Deformable Patch-based Transformer for Visual Recognition](https://mlsanity.com/paper/2107.14467/) (2021)
- [Review on Panoramic Imaging and Its Applications in Scene Understanding](https://mlsanity.com/paper/2205.05570/) (2022)
- [ACDNet: Adaptively Combined Dilated Convolution for Monocular Panorama Depth Estimation](https://mlsanity.com/paper/2112.14440/) (2021)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [DensePASS: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange](https://mlsanity.com/paper/2108.06383/) (2021)
- [Eliminating the Blind Spot: Adapting 3D Object Detection and Monocular Depth Estimation to 360deg Panoramic Imagery](https://mlsanity.com/paper/1808.06253/) (2018)
- [Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach](https://mlsanity.com/paper/1710.10755/) (2017)
- [DS-PASS: Detail-Sensitive Panoramic Annular Semantic Segmentation through SwaftNet for Surrounding Sensing](https://mlsanity.com/paper/1909.07721/) (2019)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [Complementary Bi-directional Feature Compression for Indoor 360deg Semantic Segmentation with Self-distillation](https://mlsanity.com/paper/2207.02437/) (2022)
- [FreDSNet: Joint Monocular Depth and Semantic Segmentation with Fast Fourier Convolutions](https://mlsanity.com/paper/2210.01595/) (2022)

## Built on by (most cited first, 26 of 26)

- [GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation](https://mlsanity.com/paper/2403.16370/) (2024)
- [OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentation](https://mlsanity.com/paper/2503.07098/) (2025)
- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [Open Panoramic Segmentation](https://mlsanity.com/paper/2407.02685/) (2024)
- [SphereDiffusion: Spherical Geometry-Aware Distortion Resilient Diffusion Model](https://mlsanity.com/paper/2403.10044/) (2024)
- [GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.09115/) (2024)
- [Multi-source Domain Adaptation for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.16469/) (2024)
- [Occlusion-Aware Seamless Segmentation](https://mlsanity.com/paper/2407.02182/) (2024)
- [Deformable Mamba for Wide Field of View Segmentation](https://mlsanity.com/paper/2411.16481/) (2024)
- [Panoramic Multimodal Semantic Occupancy Prediction for Quadruped Robots](https://mlsanity.com/paper/2603.13108/) (2026)
- [Unlocking Constraints: Source-Free Occlusion-Aware Seamless Segmentation](https://mlsanity.com/paper/2506.21198/) (2025)
- [OmniTrack++: Omnidirectional Multi-Object Tracking by Learning Large-FoV Trajectory Feedback](https://mlsanity.com/paper/2511.00510/) (2025)
- [PanoSAMic: Panoramic Image Segmentation from SAM Feature Encoding and Dual View Fusion](https://mlsanity.com/paper/2601.07447/) (2026)
- [Elite360M: Efficient 360 Multi-task Learning via Bi-projection Fusion and Cross-task Collaboration](https://mlsanity.com/paper/2408.09336/) (2024)
- [Panoramic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2505.03539/) (2025)
- [SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction](https://mlsanity.com/paper/2602.22867/) (2026)
- [O3N: Omnidirectional Open-Vocabulary Occupancy Prediction for Embodied Intelligent Robotics](https://mlsanity.com/paper/2603.12144/) (2026)
- [360-Degree Full-view Image Segmentation by Spherical Convolution compatible with Large-scale Planar Pre-trained Models](https://mlsanity.com/paper/2507.09216/) (2025)
- [JOPP-3D: Joint Open Vocabulary Semantic Segmentation on Point Clouds and Panoramas](https://mlsanity.com/paper/2603.06168/) (2026)
- [Denoise and Align: Towards Source-Free UDA for Robust Panoramic Semantic Segmentation](https://mlsanity.com/paper/2603.25131/) (2026)
- [Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling](https://mlsanity.com/paper/2606.27745/) (2026)
- [AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2608.29081/) (2026)
- [PanoSeg3R: Feed-Forward 3D Semantic Segmentation for Panoramic Images with an Automatic Data Curation Pipeline](https://mlsanity.com/paper/2609.22687/) (2026)
- [SAP: Segment Any 4K Panorama](https://mlsanity.com/paper/2603.12759/) (2026)
- [Gauge-Equivariant Attention for Rotation-Stable 360^circ Scene Understanding](https://mlsanity.com/paper/2609.31755/) (2026)
- [EmbPASS: Towards Cross-Embodiment Open Panoramic Segmentation](https://mlsanity.com/paper/2610.03248/) (2026)

## Abstract (arXiv)

As an important and challenging problem in computer vision, PAnoramic Semantic Segmentation (PASS) gives complete scene perception based on an ultra-wide angle of view. Usually, prevalent PASS methods with 2D panoramic image input focus on solving image distortions but lack consideration of the 3D properties of original 360circ data. Therefore, their performance will drop a lot when inputting panoramic images with the 3D disturbance. To be more robust to 3D disturbance, we propose our Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation (SGAT4PASS), considering 3D spherical geometry knowledge. Specifically, a spherical geometry-aware framework is proposed for PASS. It includes three modules, i.e., spherical geometry-aware image projection, spherical deformable patch embedding, and a panorama-aware loss, which takes input images with 3D disturbance into account, adds a spherical geometry-aware constraint on the existing deformable patch embedding, and indicates the pixel density of original 360circ data, respectively. Experimental results on Stanford2D3D Panoramic datasets show that SGAT4PASS significantly improves performance and robustness, with approximately a 2% increase in mIoU, and when small 3D disturbances occur in the data, the stability of our performance is improved by an order of magnitude. Our code and supplementary material are available at https://github.com/TencentARC/SGAT4PASS.