# Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks

arXiv: https://arxiv.org/abs/2303.15710 · HTML: https://mlsanity.com/paper/2303.15710/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 65 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [Probabilistic End-to-end Noise Correction for Learning with Noisy Labels](https://mlsanity.com/paper/1903.07788/) (2019)
- [UnitBox: An Advanced Object Detection Network](https://mlsanity.com/paper/1608.01471/) (2016)
- [Bayesian Loss for Crowd Count Estimation with Point Supervision](https://mlsanity.com/paper/1908.03684/) (2019)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)
- [ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation](https://mlsanity.com/paper/1905.10089/) (2019)
- [UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders](https://mlsanity.com/paper/2004.05763/) (2020)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [Dual Residual Networks Leveraging the Potential of Paired Operations for Image Restoration](https://mlsanity.com/paper/1903.08817/) (2019)
- [Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.06227/) (2020)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [RGB-T Image Saliency Detection via Collaborative Graph Learning](https://mlsanity.com/paper/1905.06741/) (2019)
- [Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting](https://mlsanity.com/paper/2012.04529/) (2020)
- [FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation](https://mlsanity.com/paper/2110.08988/) (2021)
- [Multi-interactive Dual-decoder for RGB-thermal Salient Object Detection](https://mlsanity.com/paper/2005.02315/) (2020)
- [HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images](https://mlsanity.com/paper/2003.04645/) (2020)
- [Deep Depth Completion from Extremely Sparse Data: A Survey](https://mlsanity.com/paper/2205.05335/) (2022)
- [Attention-based Adaptive Selection of Operations for Image Restoration in the Presence of Unknown Combined Distortions](https://mlsanity.com/paper/1812.00733/) (2018)
- [EfficientFCN: Holistically-guided Decoding for Semantic Segmentation](https://mlsanity.com/paper/2008.10487/) (2020)

## Built on by (most cited first, 30 of 41)

- [Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2404.04256/) (2024)
- [Fusion-Mamba for Cross-modality Object Detection](https://mlsanity.com/paper/2404.09146/) (2024)
- [StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation](https://mlsanity.com/paper/2408.01343/) (2024)
- [Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning](https://mlsanity.com/paper/2312.00360/) (2023)
- [Caltech Aerial RGB-Thermal Dataset in the Wild](https://mlsanity.com/paper/2403.08997/) (2024)
- [RGB-X Object Detection via Scene-Specific Fusion Modules](https://mlsanity.com/paper/2310.19372/) (2023)
- [WaveMamba: Wavelet-Driven Mamba Fusion for RGB-Infrared Object Detection](https://mlsanity.com/paper/2507.18173/) (2025)
- [MMSFormer: Multimodal Transformer for Material and Semantic Segmentation](https://mlsanity.com/paper/2309.04001/) (2023)
- [HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusion](https://mlsanity.com/paper/2404.03527/) (2024)
- [ShareCMP: Polarization-Aware RGB-P Semantic Segmentation](https://mlsanity.com/paper/2312.03430/) (2023)
- [Multispectral State-Space Feature Fusion: Bridging Shared and Cross-Parametric Interactions for Object Detection](https://mlsanity.com/paper/2507.14643/) (2025)
- [Invisible Gas Detection: An RGB-Thermal Cross Attention Network and A New Benchmark](https://mlsanity.com/paper/2403.17712/) (2024)
- [MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning](https://mlsanity.com/paper/2508.10133/) (2025)
- [ShadowSense: Unsupervised Domain Adaptation and Feature Fusion for Shadow-Agnostic Tree Crown Detection from RGB-Thermal Drone Imagery](https://mlsanity.com/paper/2310.16212/) (2023)
- [Seeing Heat with Color -- RGB-Only Wildfire Temperature Inference from SAM-Guided Multimodal Distillation using Radiometric Ground Truth](https://mlsanity.com/paper/2505.01638/) (2025)
- [Spectral-Aware Global Fusion for RGB-Thermal Semantic Segmentation](https://mlsanity.com/paper/2505.15491/) (2025)
- [Cross-modal State Space Modeling for Real-time RGB-thermal Wild Scene Semantic Segmentation](https://mlsanity.com/paper/2506.17869/) (2025)
- [IRDFusion: Iterative Relation-Map Difference guided Feature Fusion for Multispectral Object Detection](https://mlsanity.com/paper/2509.09085/) (2025)
- [CSFNet: A Cosine Similarity Fusion Network for Real-Time RGB-X Semantic Segmentation of Driving Scenes](https://mlsanity.com/paper/2407.01328/) (2024)
- [FusionSAM: Visual Multi-Modal Learning with Segment Anything](https://mlsanity.com/paper/2408.13980/) (2024)
- [IV-tuning: Parameter-Efficient Transfer Learning for Infrared-Visible Tasks](https://mlsanity.com/paper/2412.16654/) (2024)
- [Rethinking Early-Fusion Strategies for Improved Multimodal Image Segmentation](https://mlsanity.com/paper/2501.10958/) (2025)
- [RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation](https://mlsanity.com/paper/2604.12319/) (2026)
- [IVGF: The Fusion-Guided Infrared and Visible General Framework](https://mlsanity.com/paper/2409.00973/) (2024)
- [OSSA: Unsupervised One-Shot Style Adaptation](https://mlsanity.com/paper/2410.00900/) (2024)
- [Open-RGBT: Open-vocabulary RGB-T Zero-shot Semantic Segmentation in Open-world Environments](https://mlsanity.com/paper/2410.06626/) (2024)
- [Aerial Flood Scene Classification Using Fine-Tuned Attention-based Architecture for Flood-Prone Countries in South Asia](https://mlsanity.com/paper/2411.00169/) (2024)
- [BIMII-Net: Brain-Inspired Multi-Iterative Interactive Network for RGB-T Road Scene Semantic Segmentation](https://mlsanity.com/paper/2503.19303/) (2025)
- [SARTM: Segment Any RGB Thermal Model with Language aided Distillation](https://mlsanity.com/paper/2505.01950/) (2025)
- [Boosting Cross-spectral Unsupervised Domain Adaptation for Thermal Semantic Segmentation](https://mlsanity.com/paper/2505.06951/) (2025)

## Abstract (arXiv)

Recently, RGB-Thermal based perception has shown significant advances. Thermal information provides useful clues when visual cameras suffer from poor lighting conditions, such as low light and fog. However, how to effectively fuse RGB images and thermal data remains an open challenge. Previous works involve naive fusion strategies such as merging them at the input, concatenating multi-modality features inside models, or applying attention to each data modality. These fusion strategies are straightforward yet insufficient. In this paper, we propose a novel fusion method named Explicit Attention-Enhanced Fusion (EAEF) that fully takes advantage of each type of data. Specifically, we consider the following cases: i) both RGB data and thermal data, ii) only one of the types of data, and iii) none of them generate discriminative features. EAEF uses one branch to enhance feature extraction for i) and iii) and the other branch to remedy insufficient representations for ii). The outputs of two branches are fused to form complementary features. As a result, the proposed fusion method outperforms state-of-the-art by 1.6% in mIoU on semantic segmentation, 3.1% in MAE on salient object detection, 2.3% in mAP on object detection, and 8.1% in MAE on crowd counting. The code is available at https://github.com/FreeformRobotics/EAEFNet.