# Learning a Depth Covariance Function

arXiv: https://arxiv.org/abs/2303.12157 · HTML: https://mlsanity.com/paper/2303.12157/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 21 Mar 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Deep Kernel Learning](https://mlsanity.com/paper/1511.02222/) (2015)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Non-Local Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2007.10042/) (2020)
- [DeepV2D: Video to Depth with Differentiable Structure from Motion](https://mlsanity.com/paper/1812.04605/) (2018)
- [BA-Net: Dense Bundle Adjustment Network](https://mlsanity.com/paper/1806.04807/) (2018)
- [Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image](https://mlsanity.com/paper/1709.07492/) (2017)
- [CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM](https://mlsanity.com/paper/1804.00874/) (2018)
- [DeepFactors: Real-Time Probabilistic Dense Monocular SLAM](https://mlsanity.com/paper/2001.05049/) (2020)
- [Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network](https://mlsanity.com/paper/1808.00150/) (2018)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [The Promises and Pitfalls of Deep Kernel Learning](https://mlsanity.com/paper/2102.12108/) (2021)
- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [Understanding Probabilistic Sparse Gaussian Process Approximations](https://mlsanity.com/paper/1606.04820/) (2016)
- [Structured Uncertainty Prediction Networks](https://mlsanity.com/paper/1802.07079/) (2018)
- [IronDepth: Iterative Refinement of Single-View Depth using Surface Normal and its Uncertainty](https://mlsanity.com/paper/2210.03676/) (2022)
- [Sparsity Agnostic Depth Completion](https://mlsanity.com/paper/2212.00790/) (2022)
- [Learning Structured Gaussians to Approximate Deep Ensembles](https://mlsanity.com/paper/2203.15485/) (2022)
- [Balanced Depth Completion between Dense Depth Inference and Sparse Range Measurements via KISS-GP](https://mlsanity.com/paper/2008.05158/) (2020)
- [RidgeSfM: Structure from Motion via Robust Pairwise Matching Under Depth Uncertainty](https://mlsanity.com/paper/2011.10359/) (2020)
- [AK: Attentive Kernel for Information Gathering](https://mlsanity.com/paper/2205.06426/) (2022)

## Built on by (most cited first, 16 of 16)

- [Gaussian Splatting SLAM](https://mlsanity.com/paper/2312.06741/) (2023)
- [3D Reconstruction with Spatial Memory](https://mlsanity.com/paper/2408.16061/) (2024)
- [MASt3R-SLAM: Real-Time Dense SLAM with 3D Reconstruction Priors](https://mlsanity.com/paper/2412.12392/) (2024)
- [MUSt3R: Multi-view Network for Stereo 3D Reconstruction](https://mlsanity.com/paper/2503.01661/) (2025)
- [Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation](https://mlsanity.com/paper/2312.13469/) (2023)
- [PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence](https://mlsanity.com/paper/2411.16877/) (2024)
- [Depth Prompting for Sensor-Agnostic Depth Estimation](https://mlsanity.com/paper/2405.11867/) (2024)
- [AMB3R: Accurate Feed-forward Metric-scale 3D Reconstruction with Backend](https://mlsanity.com/paper/2511.20343/) (2025)
- [MAC-VO: Metrics-aware Covariance for Learning-based Stereo Visual Odometry](https://mlsanity.com/paper/2409.09479/) (2024)
- [SuperPrimitive: Scene Reconstruction at a Primitive Level](https://mlsanity.com/paper/2312.05889/) (2023)
- [COMO: Compact Mapping and Odometry](https://mlsanity.com/paper/2404.03531/) (2024)
- [SceneFactory: A Workflow-centric and Unified Framework for Incremental Scene Modeling](https://mlsanity.com/paper/2405.07847/) (2024)
- [UniQueR: Unified Query-based Feedforward 3D Reconstruction](https://mlsanity.com/paper/2603.22851/) (2026)
- [GI-SLAM: Gaussian-Inertial SLAM](https://mlsanity.com/paper/2503.18275/) (2025)
- [UDGS-SLAM : UniDepth Assisted Gaussian Splatting for Monocular SLAM](https://mlsanity.com/paper/2409.00362/) (2024)
- [BLASt3R: Bundle Adjustment of Any Image Set with Multi-View Matching and Monocular Priors](https://mlsanity.com/paper/2609.05210/) (2026)

## Abstract (arXiv)

We propose learning a depth covariance function with applications to geometric vision tasks. Given RGB images as input, the covariance function can be flexibly used to define priors over depth functions, predictive distributions given observations, and methods for active point selection. We leverage these techniques for a selection of downstream tasks: depth completion, bundle adjustment, and monocular dense visual odometry.