# Unsupervised Learning of Depth Estimation and Visual Odometry for Sparse Light Field Cameras

arXiv: https://arxiv.org/abs/2103.11322 · HTML: https://mlsanity.com/paper/2103.11322/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 21 Mar 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue](https://mlsanity.com/paper/1603.04992/) (2016)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [Learning-Based View Synthesis for Light Field Cameras](https://mlsanity.com/paper/1609.02974/) (2016)
- [SfM-Net: Learning of Structure and Motion from Video](https://mlsanity.com/paper/1704.07804/) (2017)
- [Visual Odometry Revisited: What Should Be Learnt?](https://mlsanity.com/paper/1909.09803/) (2019)
- [EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images](https://mlsanity.com/paper/1804.02379/) (2018)
- [A 4D Light-Field Dataset and CNN Architectures for Material Recognition](https://mlsanity.com/paper/1608.06985/) (2016)
- [Learning to Think Outside the Box: Wide-Baseline Light Field Depth Estimation with EPI-Shift](https://mlsanity.com/paper/1909.09059/) (2019)
- [ENG: End-to-end Neural Geometry for Robust Depth and Pose Estimation using CNNs](https://mlsanity.com/paper/1807.05705/) (2018)

## Built on by (most cited first, 3 of 3)

- [NOCaL: Calibration-Free Semi-Supervised Learning of Odometry and Camera Intrinsics](https://mlsanity.com/paper/2210.07435/) (2022)
- [Arbitrary Volumetric Refocusing of Dense and Sparse Light Fields](https://mlsanity.com/paper/2502.19238/) (2025)
- [PRISM-VO: Scale-Aware Visual Odometry Using Photometric Plenoptic Bundle Adjustment](https://mlsanity.com/paper/2607.00176/) (2026)

## Abstract (arXiv)

While an exciting diversity of new imaging devices is emerging that could dramatically improve robotic perception, the challenges of calibrating and interpreting these cameras have limited their uptake in the robotics community. In this work we generalise techniques from unsupervised learning to allow a robot to autonomously interpret new kinds of cameras. We consider emerging sparse light field (LF) cameras, which capture a subset of the 4D LF function describing the set of light rays passing through a plane. We introduce a generalised encoding of sparse LFs that allows unsupervised learning of odometry and depth. We demonstrate the proposed approach outperforming monocular and conventional techniques for dealing with 4D imagery, yielding more accurate odometry and depth maps and delivering these with metric scale. We anticipate our technique to generalise to a broad class of LF and sparse LF cameras, and to enable unsupervised recalibration for coping with shifts in camera behaviour over the lifetime of a robot. This work represents a first step toward streamlining the integration of new kinds of imaging devices in robotics applications.