# OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusion

arXiv: https://arxiv.org/abs/2203.00838 · HTML: https://mlsanity.com/paper/2203.00838/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [HoHoNet: 360 Indoor Holistic Understanding with Latent Horizontal Features](https://mlsanity.com/paper/2011.11498/) (2020)
- [OmniDepth: Dense Depth Estimation for Indoors Spherical Panoramas](https://mlsanity.com/paper/1807.09620/) (2018)
- [UniFuse: Unidirectional Fusion for 360circ Panorama Depth Estimation](https://mlsanity.com/paper/2102.03550/) (2021)
- [Tangent Images for Mitigating Spherical Distortion](https://mlsanity.com/paper/1912.09390/) (2019)
- [Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.03650/) (2016)
- [Cube Padding for Weakly-Supervised Saliency Prediction in 360deg Videos](https://mlsanity.com/paper/1806.01320/) (2018)
- [Kernel Transformer Networks for Compact Spherical Convolution](https://mlsanity.com/paper/1812.03115/) (2018)
- [Corners for Layout: End-to-End Layout Recovery from 360 Images](https://mlsanity.com/paper/1903.08094/) (2019)
- [Spherical View Synthesis for Self-Supervised 360 Depth Estimation](https://mlsanity.com/paper/1909.08112/) (2019)
- [Depth Estimation from Monocular Images and Sparse Radar Data](https://mlsanity.com/paper/2010.00058/) (2020)
- [Self-Supervised Learning of Depth and Camera Motion from 360deg Videos](https://mlsanity.com/paper/1811.05304/) (2018)
- [Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Network](https://mlsanity.com/paper/1907.00939/) (2019)
- [Pano2Vid: Automatic Cinematography for Watching 360circ Videos](https://mlsanity.com/paper/1612.02335/) (2016)

## Built on by (most cited first, 30 of 61)

- [Both Style and Distortion Matter: Dual-Path Unsupervised Domain Adaptation for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2303.14360/) (2023)
- [Review on Panoramic Imaging and Its Applications in Scene Understanding](https://mlsanity.com/paper/2205.05570/) (2022)
- [Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera](https://mlsanity.com/paper/2501.02464/) (2025)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation](https://mlsanity.com/paper/2406.12849/) (2024)
- [HRDFuse: Monocular 360degDepth Estimation by Collaboratively Learning Holistic-with-Regional Depth Distributions](https://mlsanity.com/paper/2303.11616/) (2023)
- [Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation](https://mlsanity.com/paper/2403.12505/) (2024)
- [Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement](https://mlsanity.com/paper/2311.16495/) (2023)
- [Elite360D: Towards Efficient 360 Depth Estimation via Semantic- and Distance-Aware Bi-Projection Fusion](https://mlsanity.com/paper/2403.16376/) (2024)
- [360SFUDA++: Towards Source-free UDA for Panoramic Segmentation by Learning Reliable Category Prototypes](https://mlsanity.com/paper/2404.16501/) (2024)
- [DA2: Depth Anything in Any Direction](https://mlsanity.com/paper/2509.26618/) (2025)
- [Depth Any Panoramas: A Foundation Model for Panoramic Depth Estimation](https://mlsanity.com/paper/2512.16913/) (2025)
- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [PanoGRF: Generalizable Spherical Radiance Fields for Wide-baseline Panoramas](https://mlsanity.com/paper/2306.01531/) (2023)
- [PanDA: Towards Panoramic Depth Anything with Unlabeled Panoramas and Mobius Spatial Augmentation](https://mlsanity.com/paper/2406.13378/) (2024)
- [PanelNet: Understanding 360 Indoor Environment via Panel Representation](https://mlsanity.com/paper/2305.09078/) (2023)
- [SphereUFormer: A U-Shaped Transformer for Spherical 360 Perception](https://mlsanity.com/paper/2412.06968/) (2024)
- [PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery](https://mlsanity.com/paper/2603.17571/) (2026)
- [High-Resolution Depth Estimation for 360-degree Panoramas through Perspective and Panoramic Depth Images Registration](https://mlsanity.com/paper/2210.10414/) (2022)
- [SGFormer: Spherical Geometry Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2404.14979/) (2024)
- [RomniStereo: Recurrent Omnidirectional Stereo Matching](https://mlsanity.com/paper/2401.04345/) (2024)
- [UniDAC: Universal Metric Depth Estimation for Any Camera](https://mlsanity.com/paper/2603.27105/) (2026)
- [Pano-NeRF: Synthesizing High Dynamic Range Novel Views with Geometry from Sparse Low Dynamic Range Panoramic Images](https://mlsanity.com/paper/2312.15942/) (2023)
- [EDM: Equirectangular Projection-Oriented Dense Kernelized Feature Matching](https://mlsanity.com/paper/2502.20685/) (2025)
- [S2Net: Accurate Panorama Depth Estimation on Spherical Surface](https://mlsanity.com/paper/2301.05845/) (2023)
- [PanoNormal: Monocular Indoor 360deg Surface Normal Estimation](https://mlsanity.com/paper/2405.18745/) (2024)
- [VGGT-360: Geometry-Consistent Zero-Shot Panoramic Depth Estimation](https://mlsanity.com/paper/2603.18943/) (2026)
- [Distortion-aware Transformer in 360deg Salient Object Detection](https://mlsanity.com/paper/2308.03359/) (2023)
- [Camera-Independent Single Image Depth Estimation from Defocus Blur](https://mlsanity.com/paper/2311.13045/) (2023)
- [Convolution kernel adaptation to calibrated fisheye](https://mlsanity.com/paper/2402.01456/) (2024)

## Abstract (arXiv)

A well-known challenge in applying deep-learning methods to omnidirectional images is spherical distortion. In dense regression tasks such as depth estimation, where structural details are required, using a vanilla CNN layer on the distorted 360 image results in undesired information loss. In this paper, we propose a 360 monocular depth estimation pipeline, OmniFusion, to tackle the spherical distortion issue. Our pipeline transforms a 360 image into less-distorted perspective patches (i.e. tangent images) to obtain patch-wise predictions via CNN, and then merge the patch-wise results for final output. To handle the discrepancy between patch-wise predictions which is a major issue affecting the merging quality, we propose a new framework with the following key components. First, we propose a geometry-aware feature fusion mechanism that combines 3D geometric features with 2D image features to compensate for the patch-wise discrepancy. Second, we employ the self-attention-based transformer architecture to conduct a global aggregation of patch-wise information, which further improves the consistency. Last, we introduce an iterative depth refinement mechanism, to further refine the estimated depth based on the more accurate geometric features. Experiments show that our method greatly mitigates the distortion issue, and achieves state-of-the-art performances on several 360 monocular depth estimation benchmark datasets.