# Instruction-Driven Fusion of Infrared-Visible Images: Tailoring for Diverse Downstream Tasks

arXiv: https://arxiv.org/abs/2411.09387 · HTML: https://mlsanity.com/paper/2411.09387/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 46 references that are in the map.
- First posted on arXiv 14 Nov 2024.

## Builds on (most cited first, 19 of 19)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Enhanced-alignment Measure for Binary Foreground Map Evaluation](https://mlsanity.com/paper/1805.10421/) (2018)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation](https://mlsanity.com/paper/2004.02147/) (2020)
- [LLVIP: A Visible-infrared Paired Dataset for Low-light Vision](https://mlsanity.com/paper/2108.10831/) (2021)
- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Images](https://mlsanity.com/paper/2304.05172/) (2023)
- [RGBT Salient Object Detection: A Large-scale Dataset and Benchmark](https://mlsanity.com/paper/2007.03262/) (2020)
- [CoCoNet: Coupled Contrastive Learning Network with Multi-level Feature Ensemble for Multi-modality Image Fusion](https://mlsanity.com/paper/2211.10960/) (2022)
- [MDLatLRR: A novel decomposition method for infrared and visible image fusion](https://mlsanity.com/paper/1811.02291/) (2018)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [An Interactively Reinforced Paradigm for Joint Infrared-Visible Image Fusion and Saliency Object Detection](https://mlsanity.com/paper/2305.09999/) (2023)
- [Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond](https://mlsanity.com/paper/2305.06720/) (2023)
- [PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation](https://mlsanity.com/paper/2308.03979/) (2023)
- [VIFB: A Visible and Infrared Image Fusion Benchmark](https://mlsanity.com/paper/2002.03322/) (2020)
- [What is a salient object? A dataset and a baseline model for salient object detection](https://mlsanity.com/paper/1412.5027/) (2014)
- [Fusion from Decomposition: A Self-Supervised Approach for Image Fusion and Beyond](https://mlsanity.com/paper/2410.12274/) (2024)

## Built on by (most cited first, 9 of 9)

- [UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservation](https://mlsanity.com/paper/2603.14214/) (2026)
- [Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approach](https://mlsanity.com/paper/2512.07170/) (2025)
- [Bridging Human Evaluation to Infrared and Visible Image Fusion](https://mlsanity.com/paper/2603.03871/) (2026)
- [Toward Real-world Infrared Image Super-Resolution: A Unified Autoregressive Framework and Benchmark Dataset](https://mlsanity.com/paper/2603.04745/) (2026)
- [Customized Fusion: A Closed-Loop Dynamic Network for Adaptive Multi-Task-Aware Infrared-Visible Image Fusion](https://mlsanity.com/paper/2604.08924/) (2026)
- [MAFS: Masked Autoencoder for Infrared-Visible Image Fusion and Semantic Segmentation](https://mlsanity.com/paper/2509.11817/) (2025)
- [Missing No More: Dictionary-Guided Cross-Modal Image Fusion under Missing Infrared](https://mlsanity.com/paper/2603.08018/) (2026)
- [Degradation-Robust Fusion: An Efficient Degradation-Aware Diffusion Framework for Multimodal Image Fusion in Arbitrary Degradation Scenarios](https://mlsanity.com/paper/2604.08922/) (2026)
- [Multi-modality Image Fusion under Adverse Weather: Mask-Guided Feature Restoration and Interaction](https://mlsanity.com/paper/2606.26812/) (2026)

## Abstract (arXiv)

The primary value of infrared and visible image fusion technology lies in applying the fusion results to downstream tasks. However, existing methods face challenges such as increased training complexity and significantly compromised performance of individual tasks when addressing multiple downstream tasks simultaneously. To tackle this, we propose Task-Oriented Adaptive Regulation (T-OAR), an adaptive mechanism specifically designed for multi-task environments. Additionally, we introduce the Task-related Dynamic Prompt Injection (T-DPI) module, which generates task-specific dynamic prompts from user-input text instructions and integrates them into target representations. This guides the feature extraction module to produce representations that are more closely aligned with the specific requirements of downstream tasks. By incorporating the T-DPI module into the T-OAR framework, our approach generates fusion images tailored to task-specific requirements without the need for separate training or task-specific weights. This not only reduces computational costs but also enhances adaptability and performance across multiple tasks. Experimental results show that our method excels in object detection, semantic segmentation, and salient object detection, demonstrating its strong adaptability, flexibility, and task specificity. This provides an efficient solution for image fusion in multi-task environments, highlighting the technology's potential across diverse applications.