# Brain Imaging Generation with Latent Diffusion Models

arXiv: https://arxiv.org/abs/2209.07162 · HTML: https://mlsanity.com/paper/2209.07162/

## Key facts

- Cited by 268 papers in the map (data through 14 Oct 2026).
- Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models.
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 15 Sep 2022.

## Placed in

- Diffusion models / [Medical imaging and segmentation](https://mlsanity.com/diffusion/medical-imaging-and-segmentation/)

## Builds on (most cited first, 23 of 23)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [An overview of deep learning in medical imaging focusing on MRI](https://mlsanity.com/paper/1811.10052/) (2018)
- [Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker](https://mlsanity.com/paper/1612.02572/) (2016)
- [Synthetic Data -- what, why and how?](https://mlsanity.com/paper/2205.03257/) (2022)
- [Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks](https://mlsanity.com/paper/1908.02498/) (2019)
- [Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN](https://mlsanity.com/paper/2008.01910/) (2020)
- [Synthetic Data: Opening the data floodgates to enable faster, more directed development of machine learning methods](https://mlsanity.com/paper/2012.04580/) (2020)
- [A Tool for Super-Resolving Multimodal Clinical MRI](https://mlsanity.com/paper/1909.01140/) (2019)

## Built on by (most cited first, 30 of 268)

- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Solving Linear Inverse Problems Provably via Posterior Sampling with Latent Diffusion Models](https://mlsanity.com/paper/2307.00619/) (2023)
- [On the Challenges and Perspectives of Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2306.05705/) (2023)
- [Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2211.10655/) (2022)
- [RoentGen: Vision-Language Foundation Model for Chest X-ray Generation](https://mlsanity.com/paper/2211.12737/) (2022)
- [Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains](https://mlsanity.com/paper/2210.04133/) (2022)
- [Differentially Private Diffusion Models Generate Useful Synthetic Images](https://mlsanity.com/paper/2302.13861/) (2023)
- [Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis](https://mlsanity.com/paper/2305.18453/) (2023)
- [Diffusion Probabilistic Models beat GANs on Medical Images](https://mlsanity.com/paper/2212.07501/) (2022)
- [Dream the Impossible: Outlier Imagination with Diffusion Models](https://mlsanity.com/paper/2309.13415/) (2023)
- [Anatomically-Controllable Medical Image Generation with Segmentation-Guided Diffusion Models](https://mlsanity.com/paper/2402.05210/) (2024)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study](https://mlsanity.com/paper/2311.00265/) (2023)
- [Advances in Medical Image Analysis with Vision Transformers: A Comprehensive Review](https://mlsanity.com/paper/2301.03505/) (2023)
- [SADM: Sequence-Aware Diffusion Model for Longitudinal Medical Image Generation](https://mlsanity.com/paper/2212.08228/) (2022)
- [WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis](https://mlsanity.com/paper/2402.19043/) (2024)
- [Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Generation](https://mlsanity.com/paper/2211.03364/) (2022)
- [Efficient Diffusion Models for Vision: A Survey](https://mlsanity.com/paper/2210.09292/) (2022)
- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions](https://mlsanity.com/paper/2402.15602/) (2024)
- [Membership Inference of Diffusion Models](https://mlsanity.com/paper/2301.09956/) (2023)
- [Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images](https://mlsanity.com/paper/2301.04802/) (2023)
- [Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D Brain MRI Synthesis](https://mlsanity.com/paper/2307.10094/) (2023)
- [MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images](https://mlsanity.com/paper/2310.03559/) (2023)
- [Enhancing Spatiotemporal Disease Progression Models via Latent Diffusion and Prior Knowledge](https://mlsanity.com/paper/2405.03328/) (2024)
- [Echo from noise: synthetic ultrasound image generation using diffusion models for real image segmentation](https://mlsanity.com/paper/2305.05424/) (2023)
- [Generating Realistic Brain MRIs via a Conditional Diffusion Probabilistic Model](https://mlsanity.com/paper/2212.08034/) (2022)
- [InverseSR: 3D Brain MRI Super-Resolution Using a Latent Diffusion Model](https://mlsanity.com/paper/2308.12465/) (2023)

## Abstract (arXiv)

Deep neural networks have brought remarkable breakthroughs in medical image analysis. However, due to their data-hungry nature, the modest dataset sizes in medical imaging projects might be hindering their full potential. Generating synthetic data provides a promising alternative, allowing to complement training datasets and conducting medical image research at a larger scale. Diffusion models recently have caught the attention of the computer vision community by producing photorealistic synthetic images. In this study, we explore using Latent Diffusion Models to generate synthetic images from high-resolution 3D brain images. We used T1w MRI images from the UK Biobank dataset (N=31,740) to train our models to learn about the probabilistic distribution of brain images, conditioned on covariables, such as age, sex, and brain structure volumes. We found that our models created realistic data, and we could use the conditioning variables to control the data generation effectively. Besides that, we created a synthetic dataset with 100,000 brain images and made it openly available to the scientific community.