# PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era

arXiv: https://arxiv.org/abs/2509.12989 · HTML: https://mlsanity.com/paper/2509.12989/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 71 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Builds on (most cited first, 30 of 44)

- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling](https://mlsanity.com/paper/1908.00222/) (2019)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features](https://mlsanity.com/paper/2011.11498/) (2020)
- [OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas](https://mlsanity.com/paper/1807.09620/) (2018)
- [PanoFormer: Panorama Transformer for Indoor 360 Depth Estimation](https://mlsanity.com/paper/2203.09283/) (2022)
- [UniFuse: Unidirectional Fusion for 360circ Panorama Depth Estimation](https://mlsanity.com/paper/2102.03550/) (2021)
- [Deep Learning for Event-based Vision: A Comprehensive Survey and Benchmarks](https://mlsanity.com/paper/2302.08890/) (2023)
- [Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation](https://mlsanity.com/paper/2203.01452/) (2022)
- [StyleLight: HDR Panorama Generation for Lighting Estimation and Editing](https://mlsanity.com/paper/2207.14811/) (2022)
- [360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Model](https://mlsanity.com/paper/2401.06578/) (2024)
- [Kernel Transformer Networks for Compact Spherical Convolution](https://mlsanity.com/paper/1812.03115/) (2018)
- [A Survey on Robotics with Foundation Models: toward Embodied AI](https://mlsanity.com/paper/2402.02385/) (2024)
- [In&Out : Diverse Image Outpainting via GAN Inversion](https://mlsanity.com/paper/2104.00675/) (2021)
- [4K4DGen: Panoramic 4D Generation at 4K Resolution](https://mlsanity.com/paper/2406.13527/) (2024)
- [UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them All](https://mlsanity.com/paper/2403.12532/) (2024)
- [Both Style and Distortion Matter: Dual-Path Unsupervised Domain Adaptation for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2303.14360/) (2023)
- [EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2304.07803/) (2023)
- [The StreetLearn Environment and Dataset](https://mlsanity.com/paper/1903.01292/) (2019)
- [360-Degree Panorama Generation from Few Unregistered NFoV Images](https://mlsanity.com/paper/2308.14686/) (2023)
- [DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion](https://mlsanity.com/paper/2410.24203/) (2024)
- [DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization](https://mlsanity.com/paper/2108.10743/) (2021)
- [Retrieval Augmented Generation and Understanding in Vision: A Survey and New Outlook](https://mlsanity.com/paper/2503.18016/) (2025)
- [SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation](https://mlsanity.com/paper/2412.11890/) (2024)
- [360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queries](https://mlsanity.com/paper/2311.17389/) (2023)
- [Shifting AI Efficiency From Model-Centric to Data-Centric Compression](https://mlsanity.com/paper/2505.19147/) (2025)
- [GoodSAM: Bridging Domain and Capacity Gaps via Segment Anything Model for Distortion-aware Panoramic Semantic Segmentation](https://mlsanity.com/paper/2403.16370/) (2024)
- [360SFUDA++: Towards Source-free UDA for Panoramic Segmentation by Learning Reliable Category Prototypes](https://mlsanity.com/paper/2404.16501/) (2024)

## Built on by (most cited first, 12 of 12)

- [Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks](https://mlsanity.com/paper/2510.25760/) (2025)
- [ODI-Bench: Can MLLMs Understand Immersive Omnidirectional Environments?](https://mlsanity.com/paper/2510.11549/) (2025)
- [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026)
- [Depth Anything in 360^circ: Towards Scale Invariance in the Wild](https://mlsanity.com/paper/2512.22819/) (2025)
- [Panoramic Affordance Prediction](https://mlsanity.com/paper/2603.15558/) (2026)
- [Rotation-Invariant Spherical Watermarking via Third-Order SO(3) Representation Coupling](https://mlsanity.com/paper/2605.26702/) (2026)
- [Panoramic Scene Understanding: A Survey from Distortion-Aware Engineering to Sphere-Native Modeling](https://mlsanity.com/paper/2606.27745/) (2026)
- [Argus: Metric Panoramic 3D Reconstruction for Indoor Scenes](https://mlsanity.com/paper/2606.30047/) (2026)
- [SAP: Segment Any 4K Panorama](https://mlsanity.com/paper/2603.12759/) (2026)
- [H-OmniStereo: Zero-Shot Omnidirectional Stereo Matching with Heading-Aligned Normal Priors](https://mlsanity.com/paper/2605.14963/) (2026)
- [OneCanvas: 3D Scene Understanding via Panoramic Reprojection](https://mlsanity.com/paper/2606.19253/) (2026)
- [OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning](https://mlsanity.com/paper/2606.30378/) (2026)

## Abstract (arXiv)

Omnidirectional vision, using 360-degree vision to understand the environment, has become increasingly critical across domains like robotics, industrial inspection, and environmental monitoring. Compared to traditional pinhole vision, omnidirectional vision provides holistic environmental awareness, significantly enhancing the completeness of scene perception and the reliability of decision-making. However, foundational research in this area has historically lagged behind traditional pinhole vision. This talk presents an emerging trend in the embodied AI era: the rapid development of omnidirectional vision, driven by growing industrial demand and academic interest. We highlight recent breakthroughs in omnidirectional generation, omnidirectional perception, omnidirectional understanding, and related datasets. Drawing on insights from both academia and industry, we propose an ideal panoramic system architecture in the embodied AI era, PANORAMA, which consists of four key subsystems. Moreover, we offer in-depth opinions related to emerging trends and cross-community impacts at the intersection of panoramic vision and embodied AI, along with the future roadmap and open challenges. This overview synthesizes state-of-the-art advancements and outlines challenges and opportunities for future research in building robust, general-purpose omnidirectional AI systems in the embodied AI era.