# Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses

arXiv: https://arxiv.org/abs/2102.07085 · HTML: https://mlsanity.com/paper/2102.07085/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 77 references that are in the map.
- First posted on arXiv 14 Feb 2021.

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network](https://mlsanity.com/paper/1609.04802/) (2016)
- [Enhanced Deep Residual Networks for Single Image Super-Resolution](https://mlsanity.com/paper/1707.02921/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Image Super-Resolution Using Very Deep Residual Channel Attention Networks](https://mlsanity.com/paper/1807.02758/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Residual Dense Network for Image Super-Resolution](https://mlsanity.com/paper/1802.08797/) (2018)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Deep Learning for Image Super-resolution: A Survey](https://mlsanity.com/paper/1902.06068/) (2019)
- [Learning Texture Transformer Network for Image Super-Resolution](https://mlsanity.com/paper/2006.04139/) (2020)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Learning-Based View Synthesis for Light Field Cameras](https://mlsanity.com/paper/1609.02974/) (2016)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Image Super-Resolution by Neural Texture Transfer](https://mlsanity.com/paper/1903.00834/) (2019)
- [CrossNet: An End-to-end Reference-based Super Resolution Network using Cross-scale Warping](https://mlsanity.com/paper/1807.10547/) (2018)
- [Spatial-Angular Interaction for Light Field Image Super-Resolution](https://mlsanity.com/paper/1912.07849/) (2019)
- [Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization](https://mlsanity.com/paper/2004.02215/) (2020)
- [A 4D Light-Field Dataset and CNN Architectures for Material Recognition](https://mlsanity.com/paper/1608.06985/) (2016)
- [Accurate Light Field Depth Estimation with Superpixel Regularization over Partially Occluded Regions](https://mlsanity.com/paper/1708.01964/) (2017)
- [Learning Light Field Angular Super-Resolution via a Geometry-Aware Network](https://mlsanity.com/paper/2002.11263/) (2020)
- [Light Field Super-Resolution Via Graph-Based Regularization](https://mlsanity.com/paper/1701.02141/) (2017)
- [Light Field Blind Motion Deblurring](https://mlsanity.com/paper/1704.05416/) (2017)
- [Light Field Video Capture Using a Learning-Based Hybrid Imaging System](https://mlsanity.com/paper/1705.02997/) (2017)
- [Deep Spatial-angular Regularization for Compressive Light Field Reconstruction over Coded Apertures](https://mlsanity.com/paper/2007.11882/) (2020)

## Built on by (most cited first, 4 of 4)

- [Probabilistic-based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising](https://mlsanity.com/paper/2306.08836/) (2023)
- [RealLiFe: Real-Time Light Field Reconstruction via Hierarchical Sparse Gradient Descent](https://mlsanity.com/paper/2307.03017/) (2023)
- [Self-supervised Learning-based Reconstruction of High-resolution 4D Light Fields](https://mlsanity.com/paper/2402.19020/) (2024)
- [Exploring Non-Local Spatial-Angular Correlations with a Hybrid Mamba-Transformer Framework for Light Field Super-Resolution](https://mlsanity.com/paper/2509.04824/) (2025)

## Abstract (arXiv)

This paper explores the problem of reconstructing high-resolution light field (LF) images from hybrid lenses, including a high-resolution camera surrounded by multiple low-resolution cameras. The performance of existing methods is still limited, as they produce either blurry results on plain textured areas or distortions around depth discontinuous boundaries. To tackle this challenge, we propose a novel end-to-end learning-based approach, which can comprehensively utilize the specific characteristics of the input from two complementary and parallel perspectives. Specifically, one module regresses a spatially consistent intermediate estimation by learning a deep multidimensional and cross-domain feature representation, while the other module warps another intermediate estimation, which maintains the high-frequency textures, by propagating the information of the high-resolution view. We finally leverage the advantages of the two intermediate estimations adaptively via the learned attention maps, leading to the final high-resolution LF image with satisfactory results on both plain textured areas and depth discontinuous boundaries. Besides, to promote the effectiveness of our method trained with simulated hybrid data on real hybrid data captured by a hybrid LF imaging system, we carefully design the network architecture and the training strategy. Extensive experiments on both real and simulated hybrid data demonstrate the significant superiority of our approach over state-of-the-art ones. To the best of our knowledge, this is the first end-to-end deep learning method for LF reconstruction from a real hybrid input. We believe our framework could potentially decrease the cost of high-resolution LF data acquisition and benefit LF data storage and transmission.