# Spectral Reconstruction and Disparity from Spatio-Spectrally Coded Light Fields via Multi-Task Deep Learning

arXiv: https://arxiv.org/abs/2103.10179 · HTML: https://mlsanity.com/paper/2103.10179/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 74 references that are in the map.
- First posted on arXiv 18 Mar 2021.

## Builds on (most cited first, 20 of 20)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Pareto Multi-Task Learning](https://mlsanity.com/paper/1912.12854/) (2019)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Adapting Auxiliary Losses Using Gradient Similarity](https://mlsanity.com/paper/1812.02224/) (2018)
- [EPINET: A Fully-Convolutional Neural Network Using Epipolar Geometry for Depth from Light Field Images](https://mlsanity.com/paper/1804.02379/) (2018)
- [A 4D Light-Field Dataset and CNN Architectures for Material Recognition](https://mlsanity.com/paper/1608.06985/) (2016)
- [Task-Aware Compressed Sensing with Generative Adversarial Networks](https://mlsanity.com/paper/1802.01284/) (2018)
- [Dual CNN Models for Unsupervised Monocular Depth Estimation](https://mlsanity.com/paper/1804.06324/) (2018)
- [A Unified Learning Based Framework for Light Field Reconstruction from Coded Projections](https://mlsanity.com/paper/1812.10532/) (2018)
- [Hyperspectral Light Field Stereo Matching](https://mlsanity.com/paper/1709.00835/) (2017)
- [Trainlets: Dictionary Learning in High Dimensions](https://mlsanity.com/paper/1602.00212/) (2016)
- [A Generative Model for Generic Light Field Reconstruction](https://mlsanity.com/paper/2005.06508/) (2020)
- [Compressive Light Field Reconstructions using Deep Learning](https://mlsanity.com/paper/1802.01722/) (2018)
- [Microlens array grid estimation, light field decoding, and calibration](https://mlsanity.com/paper/1912.13298/) (2019)
- [VommaNet: an End-to-End Network for Disparity Estimation from Reflective and Texture-less Light Field Images](https://mlsanity.com/paper/1811.07124/) (2018)

## Built on by (most cited first, 1 of 1)

- [Multidimensional Compressed Sensing for Spectral Light Field Imaging](https://mlsanity.com/paper/2405.00027/) (2024)

## Abstract (arXiv)

We present a novel method to reconstruct a spectral central view and its aligned disparity map from spatio-spectrally coded light fields. Since we do not reconstruct an intermediate full light field from the coded measurement, we refer to this as principal reconstruction. The coded light fields correspond to those captured by a light field camera in the unfocused design with a spectrally coded microlens array. In this application, the spectrally coded light field camera can be interpreted as a single-shot spectral depth camera. We investigate several multi-task deep learning methods and propose a new auxiliary loss-based training strategy to enhance the reconstruction performance. The results are evaluated using a synthetic as well as a new real-world spectral light field dataset that we captured using a custom-built camera. The results are compared to state-of-the art compressed sensing reconstruction and disparity estimation. We achieve a high reconstruction quality for both synthetic and real-world coded light fields. The disparity estimation quality is on par with or even outperforms state-of-the-art disparity estimation from uncoded RGB light fields.