mlsanityData to 14 Oct 2026

Image Fusion via Vision-Language Model

arXiv 2402.02235 ↗First posted 3 Feb 2024cs.CVcited by 40

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Image fusion integrates essential information from multiple images into a single composite, enhancing structures, textures, and refining imperfections. Existing methods predominantly focus on pixel-level and semantic visual features for recognition, but often overlook the deeper text-level semantic information beyond vision. Therefore, we introduce a novel fusion paradigm named image Fusion via vIsion-Language Model (FILM), for the first time, utilizing explicit textual information from source images to guide the fusion process. Specifically, FILM generates semantic prompts from images and inputs them into ChatGPT for comprehensive textual descriptions. These descriptions are fused within the textual domain and guide the visual information fusion, enhancing feature extraction and contextual understanding, directed by textual semantic information via cross-attention. FILM has shown promising results in four image fusion tasks: infrared-visible, medical, multi-exposure, and multi-focus image fusion. We also propose a vision-language dataset containing ChatGPT-generated paragraph descriptions for the eight image fusion datasets across four fusion tasks, facilitating future research in vision-language model-based image fusion. Code and dataset are available at https://github.com/Zhaozixiang1228/IF-FILM.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 84 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

34 of 84 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  2. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Segment Anythingcited by 8,234
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2021Restormer: Efficient Transformer for High-Resolution Image Restorationcited by 1,627
  12. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
20 more
  1. 2022Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detectioncited by 317
  2. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  3. 2022CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusioncited by 223
  4. 2018DenseFuse: A Fusion Approach to Infrared and Visible Imagescited by 180
  5. 2023DDFM: Denoising Diffusion Model for Multi-Modality Image Fusioncited by 153
  6. 2023Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentationcited by 129
  7. 2023LRRNet: A Novel Representation Learning Guided Fusion Network for Infrared and Visible Imagescited by 87
  8. 2022Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registrationcited by 80
  9. 2019Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusioncited by 79
  10. 2017DeepFuse: A Deep Unsupervised Approach for Exposure Fusion with Extreme Exposure Image Pairscited by 78
  11. 2020DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusioncited by 73
  12. 2021Discrete Cosine Transform Network for Guided Depth Map Super-Resolutioncited by 71
  13. 2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
  14. 2021Deep Gradient Projection Networks for Pan-sharpeningcited by 44
  15. 2013Medical Image Fusion: A survey of the state of the artcited by 39
  16. 2020Benchmarking and Comparing Multi-exposure Image Fusion Algorithmscited by 33
  17. 2021TransMEF: A Transformer-Based Multi-Exposure Image Fusion Framework using Self-Supervised Multi-Task Learningcited by 32
  18. 2023Mutual-Guided Dynamic Network for Image Fusioncited by 14
  19. 2020Towards Reducing Severe Defocus Spread Effects for Multi-Focus Image Fusion via an Optimization Based Strategycited by 9
  20. 2019An α-Matte Boundary Defocus Model Based Cascaded Network for Multi-focus Image Fusioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 23; 2026: 14.

40 papers.

Sort:

By area:

  1. 2025Infrared and Visible Image Fusion: From Data Compatibility to Task Adaptioncited by 46
  2. 2025Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyondcited by 35
  3. 2025One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusioncited by 32
  4. 2025UltraFusion: Ultra High Dynamic Imaging using Exposure Fusioncited by 21
  5. 2025Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspectivecited by 14
  6. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  7. 2025MMAIF: Multi-task and Multi-degradation All-in-One for Image Fusion with Language Guidancecited by 9
  8. 2025A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarkingcited by 9
  9. 2025Fusion Meets Diverse Conditions: A High-diversity Benchmark and Baseline for UAV-based Multimodal Object Detection with Condition Cuescited by 6
  10. 2024Deep Learning for Spatio-Temporal Fusion in Land Surface Temperature Estimation: A Comprehensive Survey, Experimental Analysis, and Future Trendscited by 4
  11. 2025Towards Unified Semantic and Controllable Image Fusion: A Diffusion Transformer Approachcited by 4
  12. 2025Retinex-MEF: Retinex-based Glare Effects Aware Unsupervised Multi-Exposure Image Fusioncited by 3
28 more
  1. 2025AWM-Fuse: Multi-Modality Image Fusion for Adverse Weather via Global and Local Text Perceptioncited by 3
  2. 2025Enhancing Infrared Vision: Progressive Prompt Fusion Network and Benchmarkcited by 3
  3. 2025Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniquescited by 2
  4. 2024A Semantic-Aware and Multi-Guided Network for Infrared-Visible Image Fusioncited by 1
  5. 2025SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosiscited by 1
  6. 2026SHARE: A Fully Unsupervised Framework for Single Hyperspectral Image Restorationcited by 1
  7. 2026MambaVF: State Space Model for Efficient Video Fusioncited by 1
  8. 2026Adding Thermal Awareness to Visual Systems in Real-Time via Distilled Diffusion Modelscited by 1
  9. 2024Infrared and Visible Image Fusion with Hierarchical Human Perceptioncited by 0
  10. 2025FusDreamer: Label-efficient Remote Sensing World Model for Multimodal Data Classificationcited by 0
  11. 2025RLBayes: a Bayesian Network Structure Learning Algorithm via Reinforcement Learning-Based Search Strategycited by 0
  12. 2025Robust Fusion Controller: Degradation-aware Image Fusion with Fine-grained Language Instructionscited by 0
  13. 2025Quaternion Sparse Decomposition for Multi-focus Color Image Fusioncited by 0
  14. 2025TeSG: Textual Semantic Guidance for Infrared and Visible Image Fusioncited by 0
  15. 2025MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusioncited by 0
  16. 2025SSVIF: Self-Supervised Segmentation-Oriented Visible and Infrared Image Fusioncited by 0
  17. 2025Multi-Grained Text-Guided Image Fusion for Multi-Exposure and Multi-Focus Scenarioscited by 0
  18. 2026Entity-Guided Multi-Task Learning for Infrared and Visible Image Fusioncited by 0
  19. 2026Equivariant Learning for Unsupervised Image Dehazingcited by 0
  20. 2026CNN-ViT Fusion with Adaptive Attention Gate for Brain Tumor MRI Classification: A Hybrid Deep Learning Modelcited by 0
  21. 2026EAPFusion: Intrinsic Evolving Auxiliary Prior Guidance for Infrared and Visible Image Fusioncited by 0
  22. 2026MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentationcited by 0
  23. 2026Unleashing the Representational Power of Fourier Shapes for Attacking Infrared Object Detectioncited by 0
  24. 2026FMRFusion: Frequency-Aware Multi-View Representation Learning for Heterogeneous Image Fusioncited by 0
  25. 2026ExpoMotion: A Large-Scale Benchmark and A Householder Projection Network for Multi-Exposure Fusioncited by 0
  26. 2026Coarse-to-fine Framework for Generative MEF via Implicit Neural Representationcited by 0
  27. 2026Current Injection Spiking Neural Network for Infrared and Visible Image Fusioncited by 0
  28. 2026Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Intermediate Fusion ViT Enables Efficient Text-Image Alignment in Diffusion Models
  2. 20243D Vision-Language Gaussian Splatting
  3. 2024DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception
  4. 2023FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions
  5. 2025On the Limitations of Vision-Language Models in Understanding Image Transforms
  6. 2025Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Model
  7. 2025MambaTrans: Multimodal Fusion Image Translation via Large Language Model Priors for Downstream Visual Tasks
  8. 2024MMFuser: Multimodal Multi-Layer Feature Fuser for Fine-Grained Vision-Language Understanding