# Towards Multimodal Depth Estimation from Light Fields

arXiv: https://arxiv.org/abs/2203.16542 · HTML: https://mlsanity.com/paper/2203.16542/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 23 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 7 of 7)

- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Uncertainty Estimates and Multi-Hypotheses Networks for Optical Flow](https://mlsanity.com/paper/1802.07095/) (2018)
- [SMD-Nets: Stereo Mixture Density Networks](https://mlsanity.com/paper/2104.03866/) (2021)
- [EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images](https://mlsanity.com/paper/1804.02379/) (2018)
- [Analysis of dropout learning regarded as ensemble learning](https://mlsanity.com/paper/1706.06859/) (2017)
- [NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis](https://mlsanity.com/paper/2003.08934/) (2020)
- [Learning to Think Outside the Box: Wide-Baseline Light Field Depth Estimation with EPI-Shift](https://mlsanity.com/paper/1909.09059/) (2019)

## Abstract (arXiv)

Light field applications, especially light field rendering and depth estimation, developed rapidly in recent years. While state-of-the-art light field rendering methods handle semi-transparent and reflective objects well, depth estimation methods either ignore these cases altogether or only deliver a weak performance. We argue that this is due current methods only considering a single "true" depth, even when multiple objects at different depths contributed to the color of a single pixel. Based on the simple idea of outputting a posterior depth distribution instead of only a single estimate, we develop and explore several different deep-learning-based approaches to the problem. Additionally, we contribute the first "multimodal light field depth dataset" that contains the depths of all objects which contribute to the color of a pixel. This allows us to supervise the multimodal depth prediction and also validate all methods by measuring the KL divergence of the predicted posteriors. With our thorough analysis and novel dataset, we aim to start a new line of depth estimation research that overcomes some of the long-standing limitations of this field.