# Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion

arXiv: https://arxiv.org/abs/1910.04066 · HTML: https://mlsanity.com/paper/1910.04066/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 30 of 79)

- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [Image Fusion via Vision-Language Model](https://mlsanity.com/paper/2402.02235/) (2024)
- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [SGNet: Structure Guided Network via Gradient-Frequency Awareness for Depth Map Super-Resolution](https://mlsanity.com/paper/2312.05799/) (2023)
- [Memory-augmented Deep Unfolding Network for Guided Image Super-resolution](https://mlsanity.com/paper/2203.04960/) (2022)
- [Task-driven Image Fusion with Learnable Fusion Loss](https://mlsanity.com/paper/2412.03240/) (2024)
- [Guided Depth Map Super-resolution: A Survey](https://mlsanity.com/paper/2302.09598/) (2023)
- [High-resolution Depth Maps Imaging via Attention-based Hierarchical Multi-modal Fusion](https://mlsanity.com/paper/2104.01530/) (2021)
- [DarkVisionNet: Low-Light Imaging via RGB-NIR Fusion with Deep Inconsistency Prior](https://mlsanity.com/paper/2303.06834/) (2023)
- [Symmetric Uncertainty-Aware Feature Transmission for Depth Super-Resolution](https://mlsanity.com/paper/2306.00386/) (2023)
- [Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration](https://mlsanity.com/paper/2404.00288/) (2024)
- [Deep Attentional Guided Image Filtering](https://mlsanity.com/paper/2112.06401/) (2021)
- [Recurrent Structure Attention Guidance for Depth Super-Resolution](https://mlsanity.com/paper/2301.13419/) (2023)
- [ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning](https://mlsanity.com/paper/2312.07943/) (2023)
- [Structure Flow-Guided Network for Real Depth Super-Resolution](https://mlsanity.com/paper/2301.13416/) (2023)
- [Mutual-Guided Dynamic Network for Image Fusion](https://mlsanity.com/paper/2308.12538/) (2023)
- [All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark](https://mlsanity.com/paper/2402.02090/) (2024)
- [DORNet: A Degradation Oriented and Regularized Network for Blind Depth Super-Resolution](https://mlsanity.com/paper/2410.11666/) (2024)
- [Towards Lightweight Super-Resolution with Dual Regression Learning](https://mlsanity.com/paper/2207.07929/) (2022)
- [Unsharp Mask Guided Filtering](https://mlsanity.com/paper/2106.01428/) (2021)
- [DuCos: Duality Constrained Depth Super-Resolution via Foundation Model](https://mlsanity.com/paper/2503.04171/) (2025)
- [A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking](https://mlsanity.com/paper/2505.19858/) (2025)
- [RGB Guided ToF Imaging System: A Survey of Deep Learning-based Methods](https://mlsanity.com/paper/2405.10357/) (2024)

## Abstract (arXiv)

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., Common and Unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion.