# Light Field Video Capture Using a Learning-Based Hybrid Imaging System

arXiv: https://arxiv.org/abs/1705.02997 · HTML: https://mlsanity.com/paper/1705.02997/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2017.

## Built on by (most cited first, 10 of 10)

- [Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses](https://mlsanity.com/paper/2102.07085/) (2021)
- [Acquiring a Dynamic Light Field through a Single-Shot Coded Image](https://mlsanity.com/paper/2204.12089/) (2022)
- [Deep Appearance Prefiltering](https://mlsanity.com/paper/2211.05932/) (2022)
- [H2-Stereo: High-Speed, High-Resolution Stereoscopic Video System](https://mlsanity.com/paper/2208.02436/) (2022)
- [Cross-Camera Deep Colorization](https://mlsanity.com/paper/2209.01211/) (2022)
- [Epipolar Focus Spectrum: A Novel Light Field Representation and Application in Dense-view Reconstruction](https://mlsanity.com/paper/2204.00193/) (2022)
- [ProbNVS: Fast Novel View Synthesis with Learned Probability-Guided Sampling](https://mlsanity.com/paper/2204.03476/) (2022)
- [Ray-Space Motion Compensation for Lenslet Plenoptic Video Coding](https://mlsanity.com/paper/2207.00522/) (2022)
- [Self-supervised Learning-based Reconstruction of High-resolution 4D Light Fields](https://mlsanity.com/paper/2402.19020/) (2024)
- [Stereo-Knowledge Distillation from dpMV to Dual Pixels for Light Field Video Reconstruction](https://mlsanity.com/paper/2405.11823/) (2024)

## Abstract (arXiv)

Light field cameras have many advantages over traditional cameras, as they allow the user to change various camera settings after capture. However, capturing light fields requires a huge bandwidth to record the data: a modern light field camera can only take three images per second. This prevents current consumer light field cameras from capturing light field videos. Temporal interpolation at such extreme scale (10x, from 3 fps to 30 fps) is infeasible as too much information will be entirely missing between adjacent frames. Instead, we develop a hybrid imaging system, adding another standard video camera to capture the temporal information. Given a 3 fps light field sequence and a standard 30 fps 2D video, our system can then generate a full light field video at 30 fps. We adopt a learning-based approach, which can be decomposed into two steps: spatio-temporal flow estimation and appearance estimation. The flow estimation propagates the angular information from the light field sequence to the 2D video, so we can warp input images to the target view. The appearance estimation then combines these warped images to output the final pixels. The whole process is trained end-to-end using convolutional neural networks. Experimental results demonstrate that our algorithm outperforms current video interpolation methods, enabling consumer light field videography, and making applications such as refocusing and parallax view generation achievable on videos for the first time.