# Modelling the Distribution of 3D Brain MRI using a 2D Slice VAE

arXiv: https://arxiv.org/abs/2007.04780 · HTML: https://mlsanity.com/paper/2007.04780/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jul 2020.

## Built on by (most cited first, 20 of 20)

- [WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis](https://mlsanity.com/paper/2402.19043/) (2024)
- [Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review](https://mlsanity.com/paper/2307.13125/) (2023)
- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [3D-StyleGAN: A Style-Based Generative Adversarial Network for Generative Modeling of Three-Dimensional Medical Images](https://mlsanity.com/paper/2107.09700/) (2021)
- [Explicitly Minimizing the Blur Error of Variational Autoencoders](https://mlsanity.com/paper/2304.05939/) (2023)
- [Echocardiography Segmentation with Enforced Temporal Consistency](https://mlsanity.com/paper/2112.02102/) (2021)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [A Survey of Emerging Applications of Diffusion Probabilistic Models in MRI](https://mlsanity.com/paper/2311.11383/) (2023)
- [Domain-Adaptive 3D Medical Image Synthesis: An Efficient Unsupervised Approach](https://mlsanity.com/paper/2207.00844/) (2022)
- [Pathology Synthesis of 3D-Consistent Cardiac MR Images using 2D VAEs and GANs](https://mlsanity.com/paper/2209.04223/) (2022)
- [Generating 3D Brain Tumor Regions in MRI using Vector-Quantization Generative Adversarial Networks](https://mlsanity.com/paper/2310.01251/) (2023)
- [Deep Quantile Regression for Uncertainty Estimation in Unsupervised and Supervised Lesion Detection](https://mlsanity.com/paper/2109.09374/) (2021)
- [3D Brain and Heart Volume Generative Models: A Survey](https://mlsanity.com/paper/2210.05952/) (2022)
- [FairSkin: Fair Diffusion for Skin Disease Image Generation](https://mlsanity.com/paper/2410.22551/) (2024)
- [Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimes](https://mlsanity.com/paper/2406.11659/) (2024)
- [Improved α-GAN architecture for generating 3D connected volumes with an application to radiosurgery treatment planning](https://mlsanity.com/paper/2207.11223/) (2022)
- [Integrating Anatomical Priors into a Causal Diffusion Model](https://mlsanity.com/paper/2509.09054/) (2025)
- [Domain Transfer Through Image-to-Image Translation for Uncertainty-Aware Prostate Cancer Classification](https://mlsanity.com/paper/2307.00479/) (2023)
- [Multi-sensor Learning Enables Information Transfer across Different Sensory Data and Augments Multi-modality Imaging](https://mlsanity.com/paper/2409.19420/) (2024)
- [MInDI-3D: Iterative Deep Learning in 3D for Sparse-view Cone Beam Computed Tomography](https://mlsanity.com/paper/2508.09616/) (2025)

## Abstract (arXiv)

Probabilistic modelling has been an essential tool in medical image analysis, especially for analyzing brain Magnetic Resonance Images (MRI). Recent deep learning techniques for estimating high-dimensional distributions, in particular Variational Autoencoders (VAEs), opened up new avenues for probabilistic modeling. Modelling of volumetric data has remained a challenge, however, because constraints on available computation and training data make it difficult effectively leverage VAEs, which are well-developed for 2D images. We propose a method to model 3D MR brain volumes distribution by combining a 2D slice VAE with a Gaussian model that captures the relationships between slices. We do so by estimating the sample mean and covariance in the latent space of the 2D model over the slice direction. This combined model lets us sample new coherent stacks of latent variables to decode into slices of a volume. We also introduce a novel evaluation method for generated volumes that quantifies how well their segmentations match those of true brain anatomy. We demonstrate that our proposed model is competitive in generating high quality volumes at high resolutions according to both traditional metrics and our proposed evaluation.