# Infrared and Visible Image Fusion with Language-Driven Loss in CLIP Embedding Space

arXiv: https://arxiv.org/abs/2402.16267 · HTML: https://mlsanity.com/paper/2402.16267/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 65 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [CLIPstyler: Image Style Transfer with a Single Text Condition](https://mlsanity.com/paper/2112.00374/) (2021)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators](https://mlsanity.com/paper/2108.00946/) (2021)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models](https://mlsanity.com/paper/2007.00328/) (2020)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection](https://mlsanity.com/paper/2305.09999/) (2023)
- [TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion](https://mlsanity.com/paper/2312.14209/) (2023)
- [SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective](https://mlsanity.com/paper/2303.09270/) (2023)

## Built on by (most cited first, 10 of 10)

- [S4Fusion: Saliency-aware Selective State Space Model for Infrared Visible Image Fusion](https://mlsanity.com/paper/2405.20881/) (2024)
- [Perceptual Region-Driven Infrared-Visible Co-Fusion for Extreme Scene Enhancement](https://mlsanity.com/paper/2512.06400/) (2025)
- [Language-Depth Navigated Thermal and Visible Image Fusion](https://mlsanity.com/paper/2503.08676/) (2025)
- [CLIP-driven rain perception: Adaptive deraining with pattern-aware network routing and mask-guided cross-attention](https://mlsanity.com/paper/2506.01366/) (2025)
- [PureCLIP-Depth: Prompt-Free and Decoder-Free Monocular Depth Estimation within CLIP Embedding Space](https://mlsanity.com/paper/2603.16238/) (2026)
- [The Solution for the GAIIC2024 RGB-TIR object detection Challenge](https://mlsanity.com/paper/2407.03872/) (2024)
- [Faster and Better: Reinforced Collaborative Distillation and Self-Learning for Infrared-Visible Image Fusion](https://mlsanity.com/paper/2509.02424/) (2025)
- [MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion](https://mlsanity.com/paper/2509.12901/) (2025)
- [DRFusion: Drift-Resilient Temporally Consistent Infrared-Visible Video Fusion](https://mlsanity.com/paper/2605.25775/) (2026)
- [Right Answer, Wrong Heat: Explanation-Aware Evaluation and Thermal-Grounded Feedback for MLLMs on Infrared Images](https://mlsanity.com/paper/2608.09145/) (2026)

## Abstract (arXiv)

Infrared-visible image fusion (IVIF) has attracted much attention owing to the highly-complementary properties of the two image modalities. Due to the lack of ground-truth fused images, the fusion output of current deep-learning based methods heavily depends on the loss functions defined mathematically. As it is hard to well mathematically define the fused image without ground truth, the performance of existing fusion methods is limited. In this paper, we propose to use natural language to express the objective of IVIF, which can avoid the explicit mathematical modeling of fusion output in current losses, and make full use of the advantage of language expression to improve the fusion performance. For this purpose, we present a comprehensive language-expressed fusion objective, and encode relevant texts into the multi-modal embedding space using CLIP. A language-driven fusion model is then constructed in the embedding space, by establishing the relationship among the embedded vectors representing the fusion objective and input image modalities. Finally, a language-driven loss is derived to make the actual IVIF aligned with the embedded language-driven fusion model via supervised training. Experiments show that our method can obtain much better fusion results than existing techniques. The code is available at https://github.com/wyhlaowang/LDFusion.