# Self-Supervised Learning of Depth and Camera Motion from 360deg Videos

arXiv: https://arxiv.org/abs/1811.05304 · HTML: https://mlsanity.com/paper/1811.05304/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2018.

## Built on by (most cited first, 30 of 40)

- [UniFuse: Unidirectional Fusion for 360circ Panorama Depth Estimation](https://mlsanity.com/paper/2102.03550/) (2021)
- [OmniFusion: 360 Monocular Depth Estimation via Geometry-Aware Fusion](https://mlsanity.com/paper/2203.00838/) (2022)
- [360MonoDepth: High-Resolution 360deg Monocular Depth Estimation](https://mlsanity.com/paper/2111.15669/) (2021)
- [LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Rendering](https://mlsanity.com/paper/2104.00568/) (2021)
- [EGformer: Equirectangular Geometry-biased Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2304.07803/) (2023)
- [Deep Learning for Omnidirectional Vision: A Survey and New Perspectives](https://mlsanity.com/paper/2205.10468/) (2022)
- [Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation](https://mlsanity.com/paper/2406.12849/) (2024)
- [BiFuse++: Self-supervised and Efficient Bi-projection Fusion for 360 Depth Estimation](https://mlsanity.com/paper/2209.02952/) (2022)
- [DA2: Depth Anything in Any Direction](https://mlsanity.com/paper/2509.26618/) (2025)
- [A Survey of Representation Learning, Optimization Strategies, and Applications for Omnidirectional Vision](https://mlsanity.com/paper/2502.10444/) (2025)
- [Improving 360 Monocular Depth Estimation via Non-local Dense Prediction Transformer and Joint Supervised and Self-supervised Learning](https://mlsanity.com/paper/2109.10563/) (2021)
- [GLPanoDepth: Global-to-Local Panoramic Depth Estimation](https://mlsanity.com/paper/2202.02796/) (2022)
- [PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery](https://mlsanity.com/paper/2603.17571/) (2026)
- [A Survey on RGB-D Datasets](https://mlsanity.com/paper/2201.05761/) (2022)
- [3D Scene Geometry Estimation from 360^circ Imagery: A Survey](https://mlsanity.com/paper/2401.09252/) (2024)
- [SGFormer: Spherical Geometry Transformer for 360 Depth Estimation](https://mlsanity.com/paper/2404.14979/) (2024)
- [360deg Optical Flow using Tangent Images](https://mlsanity.com/paper/2112.14331/) (2021)
- [SphereDepth: Panorama Depth Estimation from Spherical Domain](https://mlsanity.com/paper/2208.13714/) (2022)
- [3D reconstruction from spherical images: A review of techniques, applications, and prospects](https://mlsanity.com/paper/2302.04495/) (2023)
- [HiMODE: A Hybrid Monocular Omnidirectional Depth Estimation Model](https://mlsanity.com/paper/2204.05007/) (2022)
- [S2Net: Accurate Panorama Depth Estimation on Spherical Surface](https://mlsanity.com/paper/2301.05845/) (2023)
- [VGGT-360: Geometry-Consistent Zero-Shot Panoramic Depth Estimation](https://mlsanity.com/paper/2603.18943/) (2026)
- [Virtual Home Staging: Inverse Rendering and Editing an Indoor Panorama under Natural Illumination](https://mlsanity.com/paper/2311.12265/) (2023)
- [MTPano: Multi-Task Panoramic Scene Understanding via Label-Free Integration of Dense Prediction Priors](https://mlsanity.com/paper/2602.05330/) (2026)
- [Surround-view Fisheye BEV-Perception for Valet Parking: Dataset, Baseline and Distortion-insensitive Multi-task Framework](https://mlsanity.com/paper/2212.04111/) (2022)
- [RPG360: Robust 360 Depth Estimation with Perspective Foundation Models and Graph Optimization](https://mlsanity.com/paper/2509.23991/) (2025)
- [OB3D: A New Dataset for Benchmarking Omnidirectional 3D Reconstruction Using Blender](https://mlsanity.com/paper/2505.20126/) (2025)
- [Genie Sim PanoWorld: An Infinite Indoor 3D World Generation Pipeline via Panoramic Scene Modeling and Simulation](https://mlsanity.com/paper/2607.26646/) (2026)
- [Monocular Spherical Depth Estimation with Explicitly Connected Weak Layout Cues](https://mlsanity.com/paper/2206.11358/) (2022)
- [Cross-Domain Synthetic-to-Real In-the-Wild Depth and Normal Estimation for 3D Scene Understanding](https://mlsanity.com/paper/2212.05040/) (2022)

## Abstract (arXiv)

As 360deg cameras become prevalent in many autonomous systems (e.g., self-driving cars and drones), efficient 360deg perception becomes more and more important. We propose a novel self-supervised learning approach for predicting the omnidirectional depth and camera motion from a 360deg video. In particular, starting from the SfMLearner, which is designed for cameras with normal field-of-view, we introduce three key features to process 360deg images efficiently. Firstly, we convert each image from equirectangular projection to cubic projection in order to avoid image distortion. In each network layer, we use Cube Padding (CP), which pads intermediate features from adjacent faces, to avoid image boundaries. Secondly, we propose a novel "spherical" photometric consistency constraint on the whole viewing sphere. In this way, no pixel will be projected outside the image boundary which typically happens in images with normal field-of-view. Finally, rather than naively estimating six independent camera motions (i.e., naively applying SfM-Learner to each face on a cube), we propose a novel camera pose consistency loss to ensure the estimated camera motions reaching consensus. To train and evaluate our approach, we collect a new PanoSUNCG dataset containing a large amount of 360deg videos with groundtruth depth and camera motion. Our approach achieves state-of-the-art depth prediction and camera motion estimation on PanoSUNCG with faster inference speed comparing to equirectangular. In real-world indoor videos, our approach can also achieve qualitatively reasonable depth prediction by acquiring model pre-trained on PanoSUNCG.