# DDFM: Denoising Diffusion Model for Multi-Modality Image Fusion

arXiv: https://arxiv.org/abs/2303.06840 · HTML: https://mlsanity.com/paper/2303.06840/

## Key facts

- Cited by 153 papers in the map (data through 14 Oct 2026).
- Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models.
- Builds on 35 of its 76 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Placed in

- Diffusion models / [Image generation, text-to-image and editing](https://mlsanity.com/diffusion/image-generation-text-to-image-and-editing/)

## Builds on (most cited first, 30 of 35)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model](https://mlsanity.com/paper/2212.00490/) (2022)
- [Tackling the Generative Learning Trilemma with Denoising Diffusion GANs](https://mlsanity.com/paper/2112.07804/) (2021)
- [Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection](https://mlsanity.com/paper/2203.16220/) (2022)
- [CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion](https://mlsanity.com/paper/2211.14461/) (2022)
- [DenseFuse: A Fusion Approach to Infrared and Visible Images](https://mlsanity.com/paper/1804.08361/) (2018)
- [Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World](https://mlsanity.com/paper/2103.01050/) (2021)
- [RFN-Nest: An end-to-end residual fusion network for infrared and visible images](https://mlsanity.com/paper/2103.04286/) (2021)
- [Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration](https://mlsanity.com/paper/2205.11876/) (2022)
- [Deep Convolutional Neural Network for Multi-modal Image Restoration and Fusion](https://mlsanity.com/paper/1910.04066/) (2019)
- [DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2003.09210/) (2020)
- [Discrete Cosine Transform Network for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2104.06977/) (2021)
- [NestFuse: An Infrared and Visible Image Fusion Architecture based on Nest Connection and Spatial/Channel Attention Models](https://mlsanity.com/paper/2007.00328/) (2020)
- [Deep Gradient Projection Networks for Pan-sharpening](https://mlsanity.com/paper/2103.04584/) (2021)
- [Medical Image Fusion: A survey of the state of the art](https://mlsanity.com/paper/1401.0166/) (2013)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Image Fusion Transformer](https://mlsanity.com/paper/2107.09011/) (2021)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)

## Built on by (most cited first, 30 of 153)

- [Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation](https://mlsanity.com/paper/2308.02097/) (2023)
- [Equivariant Multi-Modality Image Fusion](https://mlsanity.com/paper/2305.11443/) (2023)
- [Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and Interactive Image Fusion](https://mlsanity.com/paper/2403.16387/) (2024)
- [Diffusion Models in Low-Level Vision: A Survey](https://mlsanity.com/paper/2406.11138/) (2024)
- [Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model](https://mlsanity.com/paper/2311.11638/) (2023)
- [Task-Customized Mixture of Adapters for General Image Fusion](https://mlsanity.com/paper/2403.12494/) (2024)
- [A Task-guided, Implicitly-searched and Meta-initialized Deep Model for Image Fusion](https://mlsanity.com/paper/2305.15862/) (2023)
- [Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption](https://mlsanity.com/paper/2501.10761/) (2025)
- [Image Restoration Through Generalized Ornstein-Uhlenbeck Bridge](https://mlsanity.com/paper/2312.10299/) (2023)
- [Image Fusion via Vision-Language Model](https://mlsanity.com/paper/2402.02235/) (2024)
- [CrossFuse: A Novel Cross Attention Mechanism based Infrared and Visible Image Fusion Approach](https://mlsanity.com/paper/2406.10581/) (2024)
- [MambaDFuse: A Mamba-based Dual-phase Model for Multi-modality Image Fusion](https://mlsanity.com/paper/2404.08406/) (2024)
- [Every SAM Drop Counts: Embracing Semantic Priors for Multi-Modality Image Fusion and Beyond](https://mlsanity.com/paper/2503.01210/) (2025)
- [DCEvo: Discriminative Cross-Dimensional Evolutionary Learning for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2503.17673/) (2025)
- [One Model for ALL: Low-Level Task Interaction Is a Key to Task-Agnostic Image Fusion](https://mlsanity.com/paper/2502.19854/) (2025)
- [Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model](https://mlsanity.com/paper/2410.23905/) (2024)
- [Video Depth without Video Models](https://mlsanity.com/paper/2411.19189/) (2024)
- [Spherical Space Feature Decomposition for Guided Depth Map Super-Resolution](https://mlsanity.com/paper/2303.08942/) (2023)
- [ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichment](https://mlsanity.com/paper/2509.17818/) (2025)
- [Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond](https://mlsanity.com/paper/2305.06720/) (2023)
- [TextFusion: Unveiling the Power of Textual Semantics for Controllable Image Fusion](https://mlsanity.com/paper/2312.14209/) (2023)
- [Task-driven Image Fusion with Learnable Fusion Loss](https://mlsanity.com/paper/2412.03240/) (2024)
- [FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients](https://mlsanity.com/paper/2401.02433/) (2023)
- [E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection](https://mlsanity.com/paper/2403.09323/) (2024)
- [From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion](https://mlsanity.com/paper/2401.00421/) (2023)
- [TFDet: Target-Aware Fusion for RGB-T Pedestrian Detection](https://mlsanity.com/paper/2305.16580/) (2023)
- [Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration](https://mlsanity.com/paper/2308.11165/) (2023)
- [Solving Inverse Problems with FLAIR](https://mlsanity.com/paper/2506.02680/) (2025)
- [TopoCellGen: Generating Histopathology Cell Topology with a Diffusion Model](https://mlsanity.com/paper/2412.06011/) (2024)
- [ReFusion: Learning Image Fusion from Reconstruction with Learnable Loss via Meta-Learning](https://mlsanity.com/paper/2312.07943/) (2023)

## Abstract (arXiv)

Multi-modality image fusion aims to combine different modalities to produce fused images that retain the complementary features of each modality, such as functional highlights and texture details. To leverage strong generative priors and address challenges such as unstable training and lack of interpretability for GAN-based generative methods, we propose a novel fusion algorithm based on the denoising diffusion probabilistic model (DDPM). The fusion task is formulated as a conditional generation problem under the DDPM sampling framework, which is further divided into an unconditional generation subproblem and a maximum likelihood subproblem. The latter is modeled in a hierarchical Bayesian manner with latent variables and inferred by the expectation-maximization (EM) algorithm. By integrating the inference solution into the diffusion sampling iteration, our method can generate high-quality fused images with natural image generative priors and cross-modality information from source images. Note that all we required is an unconditional pre-trained generative model, and no fine-tuning is needed. Our extensive experiments indicate that our approach yields promising fusion results in infrared-visible image fusion and medical image fusion. The code is available at urlhttps://github.com/Zhaozixiang1228/MMIF-DDFM.