# Multimodal Deep Unfolding for Guided Image Super-Resolution

arXiv: https://arxiv.org/abs/2001.07575 · HTML: https://mlsanity.com/paper/2001.07575/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2020.

## Built on by (most cited first, 13 of 13)

- [Memory-augmented Deep Unfolding Network for Guided Image Super-resolution](https://mlsanity.com/paper/2203.04960/) (2022)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [Unsharp Mask Guided Filtering](https://mlsanity.com/paper/2106.01428/) (2021)
- [Infrared Image Super-Resolution: Systematic Review, and Future Trends](https://mlsanity.com/paper/2212.12322/) (2022)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)
- [Deep Unfolding Multi-modal Image Fusion Network via Attribution Analysis](https://mlsanity.com/paper/2502.01467/) (2025)
- [Decoupling Fine Detail and Global Geometry for Compressed Depth Map Super-Resolution](https://mlsanity.com/paper/2411.03239/) (2024)
- [Learning Efficient and Effective Trajectories for Differential Equation-based Image Restoration](https://mlsanity.com/paper/2410.04811/) (2024)
- [Photothermal-SR-Net: A Customized Deep Unfolding Neural Network for Photothermal Super Resolution Imaging](https://mlsanity.com/paper/2104.10563/) (2021)
- [Working memory inspired hierarchical video decomposition with transformative representations](https://mlsanity.com/paper/2204.10105/) (2022)
- [CPCANet: Deep Unfolding Common Principal Component Analysis for Domain Generalization](https://mlsanity.com/paper/2605.05136/) (2026)
- [LargeMvC-Net: Anchor-based Deep Unfolding Network for Large-scale Multi-view Clustering](https://mlsanity.com/paper/2507.20980/) (2025)
- [Unsupervised Unfolded rPCA (U2-rPCA): Deep Interpretable Clutter Filtering for Ultrasound Microvascular Imaging](https://mlsanity.com/paper/2510.00660/) (2025)

## Abstract (arXiv)

The reconstruction of a high resolution image given a low resolution observation is an ill-posed inverse problem in imaging. Deep learning methods rely on training data to learn an end-to-end mapping from a low-resolution input to a high-resolution output. Unlike existing deep multimodal models that do not incorporate domain knowledge about the problem, we propose a multimodal deep learning design that incorporates sparse priors and allows the effective integration of information from another image modality into the network architecture. Our solution relies on a novel deep unfolding operator, performing steps similar to an iterative algorithm for convolutional sparse coding with side information; therefore, the proposed neural network is interpretable by design. The deep unfolding architecture is used as a core component of a multimodal framework for guided image super-resolution. An alternative multimodal design is investigated by employing residual learning to improve the training efficiency. The presented multimodal approach is applied to super-resolution of near-infrared and multi-spectral images as well as depth upsampling using RGB images as side information. Experimental results show that our model outperforms state-of-the-art methods.