# A Unified Framework for Multimodal Image Reconstruction and Synthesis using Denoising Diffusion Models

arXiv: https://arxiv.org/abs/2602.08249 · HTML: https://mlsanity.com/paper/2602.08249/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 27 of its 40 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Diffusion Posterior Sampling for General Noisy Inverse Problems](https://mlsanity.com/paper/2209.14687/) (2022)
- [Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing](https://mlsanity.com/paper/1912.10557/) (2019)
- [Deep Learning Techniques for Inverse Problems in Imaging](https://mlsanity.com/paper/2005.06001/) (2020)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [Low-Dose CT with a Residual Encoder-Decoder Convolutional Neural Network (RED-CNN)](https://mlsanity.com/paper/1702.00288/) (2017)
- [Unsupervised Medical Image Translation with Adversarial Diffusion Models](https://mlsanity.com/paper/2207.08208/) (2022)
- [ResViT: Residual vision transformers for multi-modal medical image synthesis](https://mlsanity.com/paper/2106.16031/) (2021)
- [Brain Tumor Segmentation with Deep Neural Networks](https://mlsanity.com/paper/1505.03540/) (2015)
- [Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems](https://mlsanity.com/paper/2303.05754/) (2023)
- [Multimodal End-to-End Autonomous Driving](https://mlsanity.com/paper/1906.03199/) (2019)
- [Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network](https://mlsanity.com/paper/1904.12200/) (2019)
- [Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging](https://mlsanity.com/paper/2203.17061/) (2022)
- [A Review of Convolutional Neural Networks for Inverse Problems in Imaging](https://mlsanity.com/paper/1710.04011/) (2017)
- [DuDoRNet: Learning a Dual-Domain Recurrent Network for Fast MRI Reconstruction with Deep T1 Prior](https://mlsanity.com/paper/2001.03799/) (2020)
- [CoLa-Diff: Conditional Latent Diffusion Model for Multi-Modal MRI Synthesis](https://mlsanity.com/paper/2303.14081/) (2023)
- [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021)
- [Multi-Contrast MRI Super-Resolution via a Multi-Stage Integration Network](https://mlsanity.com/paper/2105.08949/) (2021)
- [One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation](https://mlsanity.com/paper/2204.13738/) (2022)
- [Rethinking Diffusion Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2404.04785/) (2024)
- [Synthetic CT Generation from MRI using 3D Transformer-based Denoising Diffusion Model](https://mlsanity.com/paper/2305.19467/) (2023)
- [DisC-Diff: Disentangled Conditional Diffusion Model for Multi-Contrast MRI Super-Resolution](https://mlsanity.com/paper/2303.13933/) (2023)
- [MRI Reconstruction with Side Information using Diffusion Models](https://mlsanity.com/paper/2303.14795/) (2023)
- [Deformation-Compensated Learning for Image Reconstruction without Ground Truth](https://mlsanity.com/paper/2107.05533/) (2021)
- [Block Coordinate Plug-and-Play Methods for Blind Inverse Problems](https://mlsanity.com/paper/2305.12672/) (2023)

## Built on by (most cited first, 1 of 1)

- [Next-Dense-Stride Prediction for Multimodal Autoregressive Visual Modeling](https://mlsanity.com/paper/2607.09892/) (2026)

## Abstract (arXiv)

Image reconstruction and image synthesis are important for handling incomplete multimodal imaging data, but existing methods require various task-specific models, complicating training and deployment workflows. We introduce Any2all, a unified framework that addresses this limitation by formulating these disparate tasks as a single virtual inpainting problem. We train a single, unconditional diffusion model on the complete multimodal data stack. This model is then adapted at inference time to ``inpaint'' all target modalities from any combination of inputs of available clean images or noisy measurements. We validated Any2all on a PET/MR/CT brain dataset. Our results show that Any2all can achieve excellent performance on both multimodal reconstruction and synthesis tasks, consistently yielding images with competitive distortion-based performance and superior perceptual quality over specialized methods.