# Single Image Depth Prediction Made Better: A Multivariate Gaussian Take

arXiv: https://arxiv.org/abs/2303.18164 · HTML: https://mlsanity.com/paper/2303.18164/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 79 references that are in the map.
- First posted on arXiv 31 Mar 2023.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Dense Depth Priors for Neural Radiance Fields from Sparse Input Views](https://mlsanity.com/paper/2112.03288/) (2021)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [On the uncertainty of self-supervised monocular depth estimation](https://mlsanity.com/paper/2005.06209/) (2020)
- [Single-Image Depth Perception in the Wild](https://mlsanity.com/paper/1604.03901/) (2016)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation](https://mlsanity.com/paper/2204.00987/) (2022)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction](https://mlsanity.com/paper/1704.03489/) (2017)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation](https://mlsanity.com/paper/2203.14211/) (2022)
- [ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation](https://mlsanity.com/paper/2012.05258/) (2020)
- [Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention](https://mlsanity.com/paper/2210.09071/) (2022)
- [Cautious Model Predictive Control using Gaussian Process Regression](https://mlsanity.com/paper/1705.10702/) (2017)
- [Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation](https://mlsanity.com/paper/1803.11029/) (2018)
- [Enhancing Photorealism Enhancement](https://mlsanity.com/paper/2105.04619/) (2021)
- [Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/1704.02157/) (2017)

## Built on by (most cited first, 14 of 14)

- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Tri-Perspective View Decomposition for Geometry-Aware Depth Completion](https://mlsanity.com/paper/2403.15008/) (2024)
- [PatchFusion: An End-to-End Tile-Based Framework for High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2312.02284/) (2023)
- [PatchRefiner: Leveraging Synthetic Data for Real-Domain High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2406.06679/) (2024)
- [Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning](https://mlsanity.com/paper/2404.03658/) (2024)
- [DCDepth: Progressive Monocular Depth Estimation in Discrete Cosine Domain](https://mlsanity.com/paper/2410.14980/) (2024)
- [Stereo Risk: A Continuous Modeling Approach to Stereo Matching](https://mlsanity.com/paper/2407.03152/) (2024)
- [PatchRefiner V2: Fast and Lightweight Real-Domain High-Resolution Metric Depth Estimation](https://mlsanity.com/paper/2501.01121/) (2025)
- [LiftFormer: Lifting and Frame Theory Based Monocular Depth Estimation Using Depth and Edge Oriented Subspace Representation](https://mlsanity.com/paper/2604.06576/) (2026)
- [Mobile Robotic Multi-View Photometric Stereo](https://mlsanity.com/paper/2502.10842/) (2025)
- [Evidential Transformers for Improved Image Retrieval](https://mlsanity.com/paper/2409.01082/) (2024)
- [Is Pre-training Applicable to the Decoder for Dense Prediction?](https://mlsanity.com/paper/2503.07637/) (2025)
- [Monocular Depth Estimation From the Perspective of Feature Restoration: A Diffusion Enhanced Depth Restoration Approach](https://mlsanity.com/paper/2604.07664/) (2026)
- [Rethinking Dense Optical Flow without Test-Time Scaling](https://mlsanity.com/paper/2605.08000/) (2026)

## Abstract (arXiv)

Neural-network-based single image depth prediction (SIDP) is a challenging task where the goal is to predict the scene's per-pixel depth at test time. Since the problem, by definition, is ill-posed, the fundamental goal is to come up with an approach that can reliably model the scene depth from a set of training examples. In the pursuit of perfect depth estimation, most existing state-of-the-art learning techniques predict a single scalar depth value per-pixel. Yet, it is well-known that the trained model has accuracy limits and can predict imprecise depth. Therefore, an SIDP approach must be mindful of the expected depth variations in the model's prediction at test time. Accordingly, we introduce an approach that performs continuous modeling of per-pixel depth, where we can predict and reason about the per-pixel depth and its distribution. To this end, we model per-pixel scene depth using a multivariate Gaussian distribution. Moreover, contrary to the existing uncertainty modeling methods -- in the same spirit, where per-pixel depth is assumed to be independent, we introduce per-pixel covariance modeling that encodes its depth dependency w.r.t all the scene points. Unfortunately, per-pixel depth covariance modeling leads to a computationally expensive continuous loss function, which we solve efficiently using the learned low-rank approximation of the overall covariance matrix. Notably, when tested on benchmark datasets such as KITTI, NYU, and SUN-RGB-D, the SIDP model obtained by optimizing our loss function shows state-of-the-art results. Our method's accuracy (named MG) is among the top on the KITTI depth-prediction benchmark leaderboard.