# Image Fusion Transformer

arXiv: https://arxiv.org/abs/2107.09011 · HTML: https://mlsanity.com/paper/2107.09011/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 56 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Axial Attention in Multidimensional Transformers](https://mlsanity.com/paper/1912.12180/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation](https://mlsanity.com/paper/2003.07853/) (2020)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models](https://mlsanity.com/paper/2007.00328/) (2020)
- [Infrared and Visible Image Fusion using a Deep Learning Framework](https://mlsanity.com/paper/1804.06992/) (2018)

## Built on by (most cited first, 24 of 24)

- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [FusionMamba: Dynamic Feature Enhancement for Multimodal Image Fusion with Mamba](https://mlsanity.com/paper/2404.09498/) (2024)
- [CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion](https://mlsanity.com/paper/2211.10960/) (2022)
- [Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption](https://mlsanity.com/paper/2501.10761/) (2025)
- [CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach](https://mlsanity.com/paper/2406.10581/) (2024)
- [A Comprehensive Survey of Mamba Architectures for Medical Image Analysis: Classification, Segmentation, Restoration and Beyond](https://mlsanity.com/paper/2410.02362/) (2024)
- [MaeFuse: Transferring Omni Features with Pretrained Masked Autoencoders for Infrared and Visible Image Fusion via Guided Training](https://mlsanity.com/paper/2404.11016/) (2024)
- [Spatial-Frequency Enhanced Mamba for Multi-Modal Image Fusion](https://mlsanity.com/paper/2511.06593/) (2025)
- [Breaking Free from Fusion Rule: A Fully Semantic-driven Infrared and Visible Image Fusion](https://mlsanity.com/paper/2211.12286/) (2022)
- [Inline Citation Classification using Peripheral Context and Time-evolving Augmentation](https://mlsanity.com/paper/2303.00344/) (2023)
- [AdaFuse: Adaptive Medical Image Fusion Based on Spatial-Frequential Cross Attention](https://mlsanity.com/paper/2310.05462/) (2023)
- [Graph Representation Learning for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2311.00291/) (2023)
- [Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for Intelligent Transportation System](https://mlsanity.com/paper/2403.16227/) (2024)
- [FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration](https://mlsanity.com/paper/2603.07667/) (2026)
- [MATCNN: Infrared and Visible Image Fusion Method Based on Multi-scale CNN with Attention Transformer](https://mlsanity.com/paper/2502.01959/) (2025)
- [TTTFusion: A Test-Time Training-Based Strategy for Multimodal Medical Image Fusion in Surgical Robots](https://mlsanity.com/paper/2504.20362/) (2025)
- [Collaborative Text-to-Image Generation via Multi-Agent Reinforcement Learning and Semantic Fusion](https://mlsanity.com/paper/2510.10633/) (2025)
- [Explicit and implicit models in infrared and visible image fusion](https://mlsanity.com/paper/2206.09581/) (2022)
- [FDCT: Frequency-Aware Decomposition and Cross-Modal Token-Alignment for Multi-Sensor Target Classification](https://mlsanity.com/paper/2503.09873/) (2025)
- [Causality-Driven Infrared and Visible Image Fusion](https://mlsanity.com/paper/2505.20830/) (2025)
- [Direction-aware multi-scale gradient loss for infrared and visible image fusion](https://mlsanity.com/paper/2510.13067/) (2025)
- [DIFF-MF: A Difference-Driven Channel-Spatial State Space Model for Multi-Modal Image Fusion](https://mlsanity.com/paper/2601.05538/) (2026)

## Abstract (arXiv)

In image fusion, images obtained from different sensors are fused to generate a single image with enhanced information. In recent years, state-of-the-art methods have adopted Convolution Neural Networks (CNNs) to encode meaningful features for image fusion. Specifically, CNN-based methods perform image fusion by fusing local features. However, they do not consider long-range dependencies that are present in the image. Transformer-based models are designed to overcome this by modeling the long-range dependencies with the help of self-attention mechanism. This motivates us to propose a novel Image Fusion Transformer (IFT) where we develop a transformer-based multi-scale fusion strategy that attends to both local and long-range information (or global context). The proposed method follows a two-stage training approach. In the first stage, we train an auto-encoder to extract deep features at multiple scales. In the second stage, multi-scale features are fused using a Spatio-Transformer (ST) fusion strategy. The ST fusion blocks are comprised of a CNN and a transformer branch which capture local and long-range features, respectively. Extensive experiments on multiple benchmark datasets show that the proposed method performs better than many competitive fusion algorithms. Furthermore, we show the effectiveness of the proposed ST fusion strategy with an ablation analysis. The source code is available at: https://github.com/Vibashan/Image-Fusion-Transformer.