# Differentiable Diffusion for Dense Depth Estimation from Multi-view Images

arXiv: https://arxiv.org/abs/2106.08917 · HTML: https://mlsanity.com/paper/2106.08917/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 43 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Differentiable Surface Splatting for Point-based Geometry Processing](https://mlsanity.com/paper/1906.04173/) (2019)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints](https://mlsanity.com/paper/1910.06727/) (2019)
- [In Defense of Classical Image Processing: Fast Depth Completion on the CPU](https://mlsanity.com/paper/1802.00036/) (2018)
- [Neural Point Cloud Rendering via Multi-Plane Projection](https://mlsanity.com/paper/1912.04645/) (2019)
- [EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images](https://mlsanity.com/paper/1804.02379/) (2018)
- [A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation](https://mlsanity.com/paper/1602.03725/) (2016)
- [Depth Coefficients for Depth Completion](https://mlsanity.com/paper/1903.05421/) (2019)
- [Estimating Depth from RGB and Sparse Sensing](https://mlsanity.com/paper/1804.02771/) (2018)
- [Accurate Light Field Depth Estimation with Superpixel Regularization over Partially Occluded Regions](https://mlsanity.com/paper/1708.01964/) (2017)
- [EPI-based Oriented Relation Networks for Light Field Depth Estimation](https://mlsanity.com/paper/2007.04538/) (2020)
- [Fast and Accurate Depth Estimation from Sparse Light Fields](https://mlsanity.com/paper/1812.06856/) (2018)
- [View-consistent 4D Light Field Depth Estimation](https://mlsanity.com/paper/2009.04065/) (2020)

## Built on by (most cited first, 4 of 4)

- [Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution](https://mlsanity.com/paper/2302.08058/) (2023)
- [Diffusion-Augmented Depth Prediction with Sparse Annotations](https://mlsanity.com/paper/2308.02283/) (2023)
- [Light Field Depth Estimation via Stitched Epipolar Plane Images](https://mlsanity.com/paper/2203.15201/) (2022)
- [RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects](https://mlsanity.com/paper/2307.15988/) (2023)

## Abstract (arXiv)

We present a method to estimate dense depth by optimizing a sparse set of points such that their diffusion into a depth map minimizes a multi-view reprojection error from RGB supervision. We optimize point positions, depths, and weights with respect to the loss by differential splatting that models points as Gaussians with analytic transmittance. Further, we develop an efficient optimization routine that can simultaneously optimize the 50k+ points required for complex scene reconstruction. We validate our routine using ground truth data and show high reconstruction quality. Then, we apply this to light field and wider baseline images via self supervision, and show improvements in both average and outlier error for depth maps diffused from inaccurate sparse points. Finally, we compare qualitative and quantitative results to image processing and deep learning methods. http://visual.cs.brown.edu/diffdiffdepth