# Infrared-Assisted Single-Stage Framework for Joint Restoration and Fusion of Visible and Infrared Images under Hazy Conditions

arXiv: https://arxiv.org/abs/2411.12586 · HTML: https://mlsanity.com/paper/2411.12586/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 64 references that are in the map.
- First posted on arXiv 16 Nov 2024.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [Vision Transformers for Single Image Dehazing](https://mlsanity.com/paper/2204.03883/) (2022)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [The 2018 PIRM Challenge on Perceptual Image Super-resolution](https://mlsanity.com/paper/1809.07517/) (2018)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion](https://mlsanity.com/paper/2211.10960/) (2022)
- [Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Models](https://mlsanity.com/paper/2301.08072/) (2023)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [Depth Information Assisted Collaborative Mutual Promotion Network for Single Image Dehazing](https://mlsanity.com/paper/2403.01105/) (2024)
- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network](https://mlsanity.com/paper/2201.10147/) (2022)
- [DC-Net: Divide-and-Conquer for Salient Object Detection](https://mlsanity.com/paper/2305.14955/) (2023)
- [CFMW: Cross-modality Fusion Mamba for Robust Object Detection under Adverse Weather](https://mlsanity.com/paper/2404.16302/) (2024)
- [Quaternion Nuclear Norm minus Frobenius Norm Minimization for color image reconstruction](https://mlsanity.com/paper/2409.07797/) (2024)
- [VIFNet: An End-to-end Visible-Infrared Fusion Network for Image Dehazing](https://mlsanity.com/paper/2404.07790/) (2024)
- [CHITNet: A Complementary to Harmonious Information Transfer Network for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2309.06118/) (2023)
- [IAIFNet: An Illumination-Aware Infrared and Visible Image Fusion Network](https://mlsanity.com/paper/2309.14997/) (2023)

## Abstract (arXiv)

Infrared and visible (IR-VIS) image fusion has gained significant attention for its broad application value. However, existing methods often neglect the complementary role of infrared image in restoring visible image features under hazy conditions. To address this, we propose a joint learning framework that utilizes infrared image for the restoration and fusion of hazy IR-VIS images. To mitigate the adverse effects of feature diversity between IR-VIS images, we introduce a prompt generation mechanism that regulates modality-specific feature incompatibility. This creates a prompt selection matrix from non-shared image information, followed by prompt embeddings generated from a prompt pool. These embeddings help generate candidate features for dehazing. We further design an infrared-assisted feature restoration mechanism that selects candidate features based on haze density, enabling simultaneous restoration and fusion within a single-stage framework. To enhance fusion quality, we construct a multi-stage prompt embedding fusion module that leverages feature supplementation from the prompt generation module. Our method effectively fuses IR-VIS images while removing haze, yielding clear, haze-free fusion results. In contrast to two-stage methods that dehaze and then fuse, our approach enables collaborative training in a single-stage framework, making the model relatively lightweight and suitable for practical deployment. Experimental results validate its effectiveness and demonstrate advantages over existing methods. The source code of the paper is available at hrefhttps://github.com/fangjiaqi0909/IASSFtextcolorbluehttps://github.com/fangjiaqi0909/IASSF