# GAN-HA: A generative adversarial network with a novel heterogeneous dual-discriminator network and a new attention-based fusion strategy for infrared and visible image fusion

arXiv: https://arxiv.org/abs/2404.15992 · HTML: https://mlsanity.com/paper/2404.15992/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 84 references that are in the map.
- First posted on arXiv 24 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Deep Learning for Generic Object Detection: A Survey](https://mlsanity.com/paper/1809.02165/) (2018)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping](https://mlsanity.com/paper/2305.11003/) (2023)
- [DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairs](https://mlsanity.com/paper/1712.07384/) (2017)
- [Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model](https://mlsanity.com/paper/2311.11638/) (2023)
- [NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models](https://mlsanity.com/paper/2007.00328/) (2020)
- [Strategic Preys Make Acute Predators: Enhancing Camouflaged Object Detectors by Generating Camouflaged Objects](https://mlsanity.com/paper/2308.03166/) (2023)
- [MDLatLRR: A novel decomposition method for infrared and visible image fusion](https://mlsanity.com/paper/1811.02291/) (2018)
- [Medical Image Fusion: A survey of the state of the art](https://mlsanity.com/paper/1401.0166/) (2013)
- [Global Attention Mechanism: Retain Information to Enhance Channel-Spatial Interactions](https://mlsanity.com/paper/2112.05561/) (2021)
- [HQG-Net: Unpaired Medical Image Enhancement with High-Quality Guidance](https://mlsanity.com/paper/2307.07829/) (2023)
- [Learning Adaptive Discriminative Correlation Filters via Temporal Consistency Preserving Spatial Feature Selection for Robust Visual Tracking](https://mlsanity.com/paper/1807.11348/) (2018)
- [Source-Free Domain Adaptive Fundus Image Segmentation with Class-Balanced Mean Teacher](https://mlsanity.com/paper/2307.09973/) (2023)
- [Joint Group Feature Selection and Discriminative Filter Learning for Robust Visual Object Tracking](https://mlsanity.com/paper/1907.13242/) (2019)
- [Consistency Regularization for Generalizable Source-free Domain Adaptation](https://mlsanity.com/paper/2308.01587/) (2023)
- [Infrared and Visible Image Fusion with ResNet and zero-phase component analysis](https://mlsanity.com/paper/1806.07119/) (2018)
- [Concealed Object Segmentation with Hierarchical Coherence Modeling](https://mlsanity.com/paper/2401.11767/) (2024)

## Abstract (arXiv)

Infrared and visible image fusion (IVIF) aims to preserve thermal radiation information from infrared images while integrating texture details from visible images. Thermal radiation information is mainly expressed through image intensities, while texture details are typically expressed through image gradients. However, existing dual-discriminator generative adversarial networks (GANs) often rely on two structurally identical discriminators for learning, which do not fully account for the distinct learning needs of infrared and visible image information. To this end, this paper proposes a novel GAN with a heterogeneous dual-discriminator network and an attention-based fusion strategy (GAN-HA). Specifically, recognizing the intrinsic differences between infrared and visible images, we propose, for the first time, a novel heterogeneous dual-discriminator network to simultaneously capture thermal radiation information and texture details. The two discriminators in this network are structurally different, including a salient discriminator for infrared images and a detailed discriminator for visible images. They are able to learn rich image intensity information and image gradient information, respectively. In addition, a new attention-based fusion strategy is designed in the generator to appropriately emphasize the learned information from different source images, thereby improving the information representation ability of the fusion result. In this way, the fused images generated by GAN-HA can more effectively maintain both the salience of thermal targets and the sharpness of textures. Extensive experiments on various public datasets demonstrate the superiority of GAN-HA over other state-of-the-art (SOTA) algorithms while showcasing its higher potential for practical applications.