# Multimodal Image Fusion based on Hybrid CNN-Transformer and Non-local Cross-modal Attention

arXiv: https://arxiv.org/abs/2210.09847 · HTML: https://mlsanity.com/paper/2210.09847/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 25 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 7 of 7)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs](https://mlsanity.com/paper/1712.07384/) (2017)
- [VIFB: A Visible and Infrared Image Fusion Benchmark](https://mlsanity.com/paper/2002.03322/) (2020)

## Built on by (most cited first, 1 of 1)

- [Reversible Efficient Diffusion for Image Fusion](https://mlsanity.com/paper/2601.20260/) (2026)

## Abstract (arXiv)

The fusion of images taken by heterogeneous sensors helps to enrich the information and improve the quality of imaging. In this article, we present a hybrid model consisting of a convolutional encoder and a Transformer-based decoder to fuse multimodal images. In the encoder, a non-local cross-modal attention block is proposed to capture both local and global dependencies of multiple source images. A branch fusion module is designed to adaptively fuse the features of the two branches. We embed a Transformer module with linear complexity in the decoder to enhance the reconstruction capability of the proposed network. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed method by comparing it with existing state-of-the-art fusion models. The source code of our work is available at https://github.com/pandayuanyu/HCFusion.