# SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis

arXiv: https://arxiv.org/abs/2505.12251 · HTML: https://mlsanity.com/paper/2505.12251/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 18 May 2025.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Large-Scale Adversarial Training for Vision-and-Language Representation Learning](https://mlsanity.com/paper/2006.06195/) (2020)
- [BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks](https://mlsanity.com/paper/2305.17100/) (2023)
- [DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion](https://mlsanity.com/paper/2303.06840/) (2023)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [HairCLIP: Design Your Hair by Text and Reference Image](https://mlsanity.com/paper/2112.05142/) (2021)
- [DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2003.09210/) (2020)
- [Image Fusion via Vision-Language Model](https://mlsanity.com/paper/2402.02235/) (2024)
- [Medical Image Fusion: A survey of the state of the art](https://mlsanity.com/paper/1401.0166/) (2013)
- [TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion](https://mlsanity.com/paper/2312.14209/) (2023)
- [One Model to Edit Them All: Free-Form Text-Driven Image Manipulation with Semantic Modulations](https://mlsanity.com/paper/2210.07883/) (2022)
- [Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignment](https://mlsanity.com/paper/2203.00242/) (2022)
- [A Semantic-based Medical Image Fusion Approach](https://mlsanity.com/paper/1906.00225/) (2019)

## Built on by (most cited first, 1 of 1)

- [MIND: Multimodal Intent-Driven Network via Diffusion Transformers for Medical Image Fusion](https://mlsanity.com/paper/2607.28565/) (2026)

## Abstract (arXiv)

Multimodal medical image fusion plays a crucial role in medical diagnosis by integrating complementary information from different modalities to enhance image readability and clinical applicability. However, existing methods mainly follow computer vision standards for feature extraction and fusion strategy formulation, overlooking the rich semantic information inherent in medical images. To address this limitation, we propose a novel semantic-guided medical image fusion approach that, for the first time, incorporates medical prior knowledge into the fusion process. Specifically, we construct a publicly available multimodal medical image-text dataset, upon which text descriptions generated by BiomedGPT are encoded and semantically aligned with image features in a high-dimensional space via a semantic interaction alignment module. During this process, a cross attention based linear transformation automatically maps the relationship between textual and visual features to facilitate comprehensive learning. The aligned features are then embedded into a text-injection module for further feature-level fusion. Unlike traditional methods, we further generate diagnostic reports from the fused images to assess the preservation of medical information. Additionally, we design a medical semantic loss function to enhance the retention of textual cues from the source images. Experimental results on test datasets demonstrate that the proposed method achieves superior performance in both qualitative and quantitative evaluations while preserving more critical medical information.