# D2UF: Deep Coded Aperture Design and Unrolling Algorithm for Compressive Spectral Image Fusion

arXiv: https://arxiv.org/abs/2205.12158 · HTML: https://mlsanity.com/paper/2205.12158/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 57 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 12 of 12)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Deep Gaussian Scale Mixture Prior for Spectral Compressive Imaging](https://mlsanity.com/paper/2103.07152/) (2021)
- [NTIRE 2020 Challenge on Spectral Reconstruction from an RGB Image](https://mlsanity.com/paper/2005.03412/) (2020)
- [Deep Coded Aperture Design: An End-to-End Approach for Computational Imaging Tasks](https://mlsanity.com/paper/2105.03390/) (2021)
- [LADMM-Net: An Unrolled Deep Network For Spectral Image Fusion From Compressive Data](https://mlsanity.com/paper/2103.00940/) (2021)
- [JR2net: A Joint Non-Linear Representation and Recovery Network for Compressive Spectral Imaging](https://mlsanity.com/paper/2205.07770/) (2022)
- [Fast Hyperspectral Image Recovery via Non-iterative Fusion of Dual-Camera Compressive Hyperspectral Imaging](https://mlsanity.com/paper/2012.15104/) (2020)
- [An Overview of Advances in Signal Processing Techniques for Classical and Quantum Wideband Synthetic Apertures](https://mlsanity.com/paper/2205.05602/) (2022)
- [Feature Fusion via Dual-resolution Compressive Measurement Matrix Analysis For Spectral Image Classification](https://mlsanity.com/paper/2009.06961/) (2020)

## Built on by (most cited first, 1 of 1)

- [Integration of Programmable Diffraction with Digital Neural Networks](https://mlsanity.com/paper/2406.10688/) (2024)

## Abstract (arXiv)

Compressive spectral imaging (CSI) has attracted significant attention since it employs synthetic apertures to codify spatial and spectral information, sensing only 2D projections of the 3D spectral image. However, these optical architectures suffer from a trade-off between the spatial and spectral resolution of the reconstructed image due to technology limitations. To overcome this issue, compressive spectral image fusion (CSIF) employs the projected measurements of two CSI architectures with different resolutions to estimate a high-spatial high-spectral resolution. This work presents the fusion of the compressive measurements of a low-spatial high-spectral resolution coded aperture snapshot spectral imager (CASSI) architecture and a high-spatial low-spectral resolution multispectral color filter array (MCFA) system. Unlike previous CSIF works, this paper proposes joint optimization of the sensing architectures and a reconstruction network in an end-to-end (E2E) manner. The trainable optical parameters are the coded aperture (CA) in the CASSI and the colored coded aperture in the MCFA system, employing a sigmoid activation function and regularization function to encourage binary values on the trainable variables for an implementation purpose. Additionally, an unrolling-based network inspired by the alternating direction method of multipliers (ADMM) optimization is formulated to address the reconstruction step and the acquisition systems design jointly. Finally, a spatial-spectral inspired loss function is employed at the end of each unrolling layer to increase the convergence of the unrolling network. The proposed method outperforms previous CSIF methods, and experimental results validate the method with real measurements.