# FuseFormer: A Transformer for Visual and Thermal Image Fusion

arXiv: https://arxiv.org/abs/2402.00971 · HTML: https://mlsanity.com/paper/2402.00971/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 50 references that are in the map.
- First posted on arXiv 1 Feb 2024.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [SwinFuse: A Residual Swin Transformer Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2204.11436/) (2022)
- [Infrared and Visible Image Fusion with ResNet and zero-phase component analysis](https://mlsanity.com/paper/1806.07119/) (2018)
- [A Dual-branch Network for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2101.09643/) (2021)
- [PPT Fusion: Pyramid Patch Transformerfor a Case Study in Image Fusion](https://mlsanity.com/paper/2107.13967/) (2021)
- [A Survey on Infrared Image and Video Sets](https://mlsanity.com/paper/2203.08581/) (2022)
- [TransFuse: A Unified Transformer-based Image Fusion Framework using Self-supervised Learning](https://mlsanity.com/paper/2201.07451/) (2022)

## Built on by (most cited first, 2 of 2)

- [Spatial-Frequency Enhanced Mamba for Multi-Modal Image Fusion](https://mlsanity.com/paper/2511.06593/) (2025)
- [Adaptive Real-Time Multi-Loss Function Optimization Using Dynamic Memory Fusion Framework: A Case Study on Breast Cancer Segmentation](https://mlsanity.com/paper/2410.19745/) (2024)

## Abstract (arXiv)

Due to the lack of a definitive ground truth for the image fusion problem, the loss functions are structured based on evaluation metrics, such as the structural similarity index measure (SSIM). However, in doing so, a bias is introduced toward the SSIM and, consequently, the input visual band image. The objective of this study is to propose a novel methodology for the image fusion problem that mitigates the limitations associated with using classical evaluation metrics as loss functions. Our approach integrates a transformer-based multi-scale fusion strategy that adeptly addresses local and global context information. This integration not only refines the individual components of the image fusion process but also significantly enhances the overall efficacy of the method. Our proposed method follows a two-stage training approach, where an auto-encoder is initially trained to extract deep features at multiple scales in the first stage. For the second stage, we integrate our fusion block and change the loss function as mentioned. The multi-scale features are fused using a combination of Convolutional Neural Networks (CNNs) and Transformers. The CNNs are utilized to capture local features, while the Transformer handles the integration of general context features. Through extensive experiments on various benchmark datasets, our proposed method, along with the novel loss function definition, demonstrates superior performance compared to other competitive fusion algorithms.