# Deep Learning for Omnidirectional Vision: A Survey and New Perspectives

arXiv: https://arxiv.org/abs/2205.10468 · HTML: https://mlsanity.com/paper/2205.10468/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 124 of its 243 references that are in the map.
- First posted on arXiv 21 May 2022.

## Builds on (most cited first, 30 of 124)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval](https://mlsanity.com/paper/2011.12172/) (2020)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization](https://mlsanity.com/paper/2204.00097/) (2022)
- [Learning to Predict Indoor Illumination from a Single Image](https://mlsanity.com/paper/1704.00090/) (2017)
- [Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching](https://mlsanity.com/paper/2005.03860/) (2020)
- [Pathdreamer: A World Model for Indoor Navigation](https://mlsanity.com/paper/2105.08756/) (2021)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization](https://mlsanity.com/paper/2103.06818/) (2021)
- [HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentation](https://mlsanity.com/paper/1901.03861/) (2019)

## Built on by (most cited first, 30 of 38)

- [360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model](https://mlsanity.com/paper/2401.06578/) (2024)
- [Both Style and Distortion Matter: Dual-Path Unsupervised Domain Adaptation for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2303.14360/) (2023)
- [Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2207.11860/) (2022)
- [Look at the Neighbor: Distortion-aware Unsupervised Domain Adaptation for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2308.05493/) (2023)
- [HRDFuse: Monocular 360degDepth Estimation by Collaboratively Learning Holistic-with-Regional Depth Distributions](https://mlsanity.com/paper/2303.11616/) (2023)
- [A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming](https://mlsanity.com/paper/2404.16038/) (2024)
- [Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation](https://mlsanity.com/paper/2403.12505/) (2024)
- [Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion](https://mlsanity.com/paper/2403.16376/) (2024)
- [GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation](https://mlsanity.com/paper/2403.16370/) (2024)
- [360SFUDA++: Towards Source-free UDA for Panoramic Segmentation by Learning Reliable Category Prototypes](https://mlsanity.com/paper/2404.16501/) (2024)
- [OmniSAM: Omnidirectional Segment Anything Model for UDA in Panoramic Semantic Segmentation](https://mlsanity.com/paper/2503.07098/) (2025)
- [OSRT: Omnidirectional Image Super-Resolution with Distortion-aware Transformer](https://mlsanity.com/paper/2302.03453/) (2023)
- [SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation](https://mlsanity.com/paper/2306.03403/) (2023)
- [Single Frame Semantic Segmentation Using Multi-Modal Spherical Images](https://mlsanity.com/paper/2308.09369/) (2023)
- [PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation](https://mlsanity.com/paper/2309.12303/) (2023)
- [Open Panoramic Segmentation](https://mlsanity.com/paper/2407.02685/) (2024)
- [Minimalist and High-Quality Panoramic Imaging with PSF-aware Transformers](https://mlsanity.com/paper/2306.12992/) (2023)
- [GoodSAM++: Bridging Domain and Capacity Gaps via Segment Anything Model for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2408.09115/) (2024)
- [Neural Brain: A Neuroscience-inspired Framework for Embodied Agents](https://mlsanity.com/paper/2505.07634/) (2025)
- [OPDN: Omnidirectional Position-aware Deformable Network for Omnidirectional Image Super-Resolution](https://mlsanity.com/paper/2304.13471/) (2023)
- [Occlusion-Aware Seamless Segmentation](https://mlsanity.com/paper/2407.02182/) (2024)
- [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026)
- [Helvipad: A Real-World Dataset for Omnidirectional Stereo Depth Estimation](https://mlsanity.com/paper/2411.18335/) (2024)
- [Seam360GS: Seamless 360deg Gaussian Splatting from Real-World Omnidirectional Images](https://mlsanity.com/paper/2508.20080/) (2025)
- [Applications of Deep Learning for Top-View Omnidirectional Imaging: A Survey](https://mlsanity.com/paper/2304.08193/) (2023)
- [Elite360M: Efficient 360 Multi-task Learning via Bi-projection Fusion and Cross-task Collaboration](https://mlsanity.com/paper/2408.09336/) (2024)
- [RealOSR: Latent Guidance Boosts Diffusion-based Real-world Omnidirectional Image Super-Resolutions](https://mlsanity.com/paper/2412.09646/) (2024)
- [Spatio-Temporal Distortion Aware Omnidirectional Video Super-Resolution](https://mlsanity.com/paper/2410.11506/) (2024)
- [SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Dense Prediction](https://mlsanity.com/paper/2602.22867/) (2026)
- [Learning High-Quality Navigation and Zooming on Omnidirectional Images in Virtual Reality](https://mlsanity.com/paper/2405.00351/) (2024)

## Abstract (arXiv)

Omnidirectional image (ODI) data is captured with a 360x180 field-of-view, which is much wider than the pinhole cameras and contains richer spatial information than the conventional planar images. Accordingly, omnidirectional vision has attracted booming attention due to its more advantageous performance in numerous applications, such as autonomous driving and virtual reality. In recent years, the availability of customer-level 360 cameras has made omnidirectional vision more popular, and the advance of deep learning (DL) has significantly sparked its research and applications. This paper presents a systematic and comprehensive review and analysis of the recent progress in DL methods for omnidirectional vision. Our work covers four main contents: (i) An introduction to the principle of omnidirectional imaging, the convolution methods on the ODI, and datasets to highlight the differences and difficulties compared with the 2D planar image data; (ii) A structural and hierarchical taxonomy of the DL methods for omnidirectional vision; (iii) A summarization of the latest novel learning strategies and applications; (iv) An insightful discussion of the challenges and open problems by highlighting the potential research directions to trigger more research in the community.