# Self-supervised Learning-based Reconstruction of High-resolution 4D Light Fields

arXiv: https://arxiv.org/abs/2402.19020 · HTML: https://mlsanity.com/paper/2402.19020/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 29 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trends](https://mlsanity.com/paper/2301.05712/) (2023)
- [Blind Image Super-Resolution: A Survey and Beyond](https://mlsanity.com/paper/2107.03055/) (2021)
- [Unsupervised Real-world Image Super Resolution via Domain-distance Aware Training](https://mlsanity.com/paper/2004.01178/) (2020)
- [Disentangling Light Fields for Super-Resolution and Disparity Estimation](https://mlsanity.com/paper/2202.10603/) (2022)
- [Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization](https://mlsanity.com/paper/2004.02215/) (2020)
- [Light Field Image Super-Resolution with Transformers](https://mlsanity.com/paper/2108.07597/) (2021)
- [Accurate Light Field Depth Estimation with Superpixel Regularization over Partially Occluded Regions](https://mlsanity.com/paper/1708.01964/) (2017)
- [Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolution](https://mlsanity.com/paper/2302.08058/) (2023)
- [Learning Light Field Angular Super-Resolution via a Geometry-Aware Network](https://mlsanity.com/paper/2002.11263/) (2020)
- [Occlusion-aware Unsupervised Learning of Depth from 4-D Light Fields](https://mlsanity.com/paper/2106.03043/) (2021)
- [Light Field Super-Resolution Via Graph-Based Regularization](https://mlsanity.com/paper/1701.02141/) (2017)
- [Light Field Video Capture Using a Learning-Based Hybrid Imaging System](https://mlsanity.com/paper/1705.02997/) (2017)
- [Real-World Light Field Image Super-Resolution via Degradation Modulation](https://mlsanity.com/paper/2206.06214/) (2022)
- [Hybrid Light Field Imaging for Improved Spatial Resolution and Depth Range](https://mlsanity.com/paper/1611.05008/) (2016)
- [Probabilistic-based Feature Embedding of 4-D Light Fields for Compressive Imaging and Denoising](https://mlsanity.com/paper/2306.08836/) (2023)
- [Light Field Reconstruction via Deep Adaptive Fusion of Hybrid Lenses](https://mlsanity.com/paper/2102.07085/) (2021)
- [Light Field Depth Estimation via Stitched Epipolar Plane Images](https://mlsanity.com/paper/2203.15201/) (2022)
- [Toward Real-World Light Field Super-Resolution](https://mlsanity.com/paper/2305.18994/) (2023)

## Abstract (arXiv)

Hand-held light field (LF) cameras often exhibit low spatial resolution due to the inherent trade-off between spatial and angular dimensions. Existing supervised learning-based LF spatial super-resolution (SR) methods, which rely on pre-defined image degradation models, struggle to overcome the domain gap between the training phase -- where LFs with natural resolution are used as ground truth -- and the inference phase, which aims to reconstruct higher-resolution LFs, especially when applied to real-world data.To address this challenge, this paper introduces a novel self-supervised learning-based method for LF spatial SR, which can produce higher spatial resolution LF images than originally captured ones without pre-defined image degradation models. The self-supervised method incorporates a hybrid LF imaging prototype, a real-world hybrid LF dataset, and a self-supervised LF spatial SR framework. The prototype makes reference image pairs between low-resolution central-view sub-aperture images and high-resolution (HR) images. The self-supervised framework consists of a well-designed LF spatial SR network with hybrid input, a central-view synthesis network with an HR-aware loss that enables side-view sub-aperture images to learn high-frequency information from the only HR central view reference image, and a backward degradation network with an epipolar-plane image gradient loss to preserve LF parallax structures. Extensive experiments on both simulated and real-world datasets demonstrate the significant superiority of our approach over state-of-the-art ones in reconstructing higher spatial resolution LF images without pre-defined degradation.