# ControlFusion: A Controllable Image Fusion Framework with Language-Vision Degradation Prompts

arXiv: https://arxiv.org/abs/2503.23356 · HTML: https://mlsanity.com/paper/2503.23356/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 56 references that are in the map.
- First posted on arXiv 30 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency](https://mlsanity.com/paper/2108.06858/) (2021)
- [Heavy Rain Image Restoration: Integrating Physics Model and Conditional Adversarial Learning](https://mlsanity.com/paper/1904.05050/) (2019)
- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [InstructIR: High-Quality Image Restoration Following Human Instructions](https://mlsanity.com/paper/2401.16468/) (2024)
- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [Controlling Vision-Language Models for Multi-Task Image Restoration](https://mlsanity.com/paper/2310.01018/) (2023)
- [AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusion](https://mlsanity.com/paper/2310.10123/) (2023)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [OneRestore: A Universal Restoration Framework for Composite Degradation](https://mlsanity.com/paper/2407.04621/) (2024)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation](https://mlsanity.com/paper/2308.03979/) (2023)
- [SPIRE: Semantic Prompt-Driven Image Restoration](https://mlsanity.com/paper/2312.11595/) (2023)
- [All-weather Multi-Modality Image Fusion: Unified Framework and 100k Benchmark](https://mlsanity.com/paper/2402.02090/) (2024)
- [Vehicle Tracking Using Surveillance with Multimodal Data Fusion](https://mlsanity.com/paper/1811.02627/) (2018)

## Built on by (most cited first, 9 of 9)

- [Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach](https://mlsanity.com/paper/2512.07170/) (2025)
- [FusionRegister: Every Infrared and Visible Image Fusion Deserves Registration](https://mlsanity.com/paper/2603.07667/) (2026)
- [Multi-Modal Image Fusion via Intervention-Stable Feature Learning](https://mlsanity.com/paper/2603.23272/) (2026)
- [Adding Thermal Awareness to Visual Systems in Real-Time via Distilled Diffusion Models](https://mlsanity.com/paper/2605.06010/) (2026)
- [Robust Fusion Controller: Degradation-aware Image Fusion with Fine-grained Language Instructions](https://mlsanity.com/paper/2504.05795/) (2025)
- [MagicFuse: Single Image Fusion for Visual and Semantic Reinforcement](https://mlsanity.com/paper/2602.01760/) (2026)
- [UMSS: Towards Unsupervised Multi-modal Semantic Segmentation](https://mlsanity.com/paper/2607.12372/) (2026)
- [When Integral Meets Decomposition: A Signal-Level Self-Supervised Feature Decompose Paradigm for Multi-Modal Image Fusion](https://mlsanity.com/paper/2609.39004/) (2026)
- [AD-SRF: Attack-Defense Semantic Reinforcement for Multi-Modal Image Fusion](https://mlsanity.com/paper/2610.12897/) (2026)

## Abstract (arXiv)

Current image fusion methods struggle to address the composite degradations encountered in real-world imaging scenarios and lack the flexibility to accommodate user-specific requirements. In response to these challenges, we propose a controllable image fusion framework with language-vision prompts, termed ControlFusion, which adaptively neutralizes composite degradations. On the one hand, we develop a degraded imaging model that integrates physical imaging mechanisms, including the Retinex theory and atmospheric scattering principle, to simulate composite degradations, thereby providing potential for addressing real-world complex degradations from the data level. On the other hand, we devise a prompt-modulated restoration and fusion network that dynamically enhances features with degradation prompts, enabling our method to accommodate composite degradation of varying levels. Specifically, considering individual variations in quality perception of users, we incorporate a text encoder to embed user-specified degradation types and severity levels as degradation prompts. We also design a spatial-frequency collaborative visual adapter that autonomously perceives degradations in source images, thus eliminating the complete dependence on user instructions. Extensive experiments demonstrate that ControlFusion outperforms SOTA fusion methods in fusion quality and degradation handling, particularly in countering real-world and compound degradations with various levels. The source code is publicly available at https://github.com/Linfeng-Tang/ControlFusion.