# l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion

arXiv: https://arxiv.org/abs/2411.04519 · HTML: https://mlsanity.com/paper/2411.04519/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 61 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 20 of 20)

- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion](https://mlsanity.com/paper/1910.04066/) (2019)
- [FISTA-Net: Learning A Fast Iterative Shrinkage Thresholding Network for Inverse Problems in Imaging](https://mlsanity.com/paper/2008.02683/) (2020)
- [CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion](https://mlsanity.com/paper/2211.10960/) (2022)
- [MDLatLRR: A novel decomposition method for infrared and visible image fusion](https://mlsanity.com/paper/1811.02291/) (2018)
- [Convolutional Neural Networks Analyzed via Convolutional Sparse Coding](https://mlsanity.com/paper/1607.08194/) (2016)
- [Quasi-hyperbolic momentum and Adam for deep learning](https://mlsanity.com/paper/1810.06801/) (2018)
- [CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach](https://mlsanity.com/paper/2406.10581/) (2024)
- [Maximal Sparsity with Deep Networks?](https://mlsanity.com/paper/1605.01636/) (2016)
- [Learned Convolutional Sparse Coding](https://mlsanity.com/paper/1711.00328/) (2017)
- [Ada-LISTA: Learned Solvers Adaptive to Varying Models](https://mlsanity.com/paper/2001.08456/) (2020)
- [Convolutional Dictionary Learning via Local Processing](https://mlsanity.com/paper/1705.03239/) (2017)
- [A Multi-scale Information Integration Framework for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2312.04328/) (2023)
- [Coupled Feature Learning for Multimodal Medical Image Fusion](https://mlsanity.com/paper/2102.08641/) (2021)

## Built on by (most cited first, 2 of 2)

- [Hyperspectral Unmixing with 3D Convolutional Sparse Coding and Projected Simplex Volume Maximization](https://mlsanity.com/paper/2512.05674/) (2025)
- [Combined Dictionary Unfolding Network with Gradient-Adaptive Fidelity for Transferable Multi-Source Fusion](https://mlsanity.com/paper/2605.00461/) (2026)

## Abstract (arXiv)

Multi-modal image fusion (MMIF) enhances the information content of the fused image by combining the unique as well as common features obtained from different modality sensor images, improving visualization, object detection, and many more tasks. In this work, we introduce an interpretable network for the MMIF task, named FNet, based on an ℓ0-regularized multi-modal convolutional sparse coding (MCSC) model. Specifically, for solving the ℓ0-regularized CSC problem, we design a learnable ℓ0-regularized sparse coding (LZSC) block in a principled manner through deep unfolding. Given different modality source images, FNet first separates the unique and common features from them using the LZSC block and then these features are combined to generate the final fused image. Additionally, we propose an ℓ0-regularized MCSC model for the inverse fusion process. Based on this model, we introduce an interpretable inverse fusion network named IFNet, which is utilized during FNet's training. Extensive experiments show that FNet achieves high-quality fusion results across eight different MMIF datasets. Furthermore, we show that FNet enhances downstream object detection textcolor[rgb] 0, 0, 0and semantic segmentation in visible-thermal image pairs. We have also visualized the intermediate results of FNet, which demonstrates the good interpretability of our network. Link for code and models: https://github.com/gargi884/FNet-MMIF.