# Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN

arXiv: https://arxiv.org/abs/2008.01910 · HTML: https://mlsanity.com/paper/2008.01910/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2020.

## Built on by (most cited first, 30 of 60)

- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis](https://mlsanity.com/paper/2305.18453/) (2023)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [WDM: 3D Wavelet Diffusion Models for High-Resolution Medical Image Synthesis](https://mlsanity.com/paper/2402.19043/) (2024)
- [MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images](https://mlsanity.com/paper/2310.03559/) (2023)
- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation](https://mlsanity.com/paper/2304.04106/) (2023)
- [Generating Realistic Brain MRIs via a Conditional Diffusion Probabilistic Model](https://mlsanity.com/paper/2212.08034/) (2022)
- [Flow Matching for Medical Image Synthesis: Bridging the Gap Between Speed and Quality](https://mlsanity.com/paper/2503.00266/) (2025)
- [Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges](https://mlsanity.com/paper/2407.00116/) (2024)
- [MAISI-v2: Accelerated 3D High-Resolution Medical Image Synthesis with Rectified Flow and Region-specific Contrastive Loss](https://mlsanity.com/paper/2508.05772/) (2025)
- [Fr'echet Radiomic Distance (FRD): A Versatile Metric for Comparing Medical Imaging Datasets](https://mlsanity.com/paper/2412.01496/) (2024)
- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [MedSegFactory: Text-Guided Generation of Medical Image-Mask Pairs](https://mlsanity.com/paper/2504.06897/) (2025)
- [Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model](https://mlsanity.com/paper/2505.04522/) (2025)
- [Metadata-Conditioned Generative Models to Synthesize Anatomically-Plausible 3D Brain MRIs](https://mlsanity.com/paper/2310.04630/) (2023)
- [Latent Causal Modeling for 3D Brain MRI Counterfactuals](https://mlsanity.com/paper/2409.05585/) (2024)
- [Lung-DDPM: Semantic Layout-guided Diffusion Models for Thoracic CT Image Synthesis](https://mlsanity.com/paper/2502.15204/) (2025)
- [From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation](https://mlsanity.com/paper/2506.00633/) (2025)
- [Memory-efficient High-resolution OCT Volume Synthesis with Cascaded Amortized Latent Diffusion Models](https://mlsanity.com/paper/2405.16516/) (2024)
- [WASABI: A Metric for Evaluating Morphometric Plausibility of Synthetic Brain MRIs](https://mlsanity.com/paper/2504.21771/) (2025)
- [3D Brain and Heart Volume Generative Models: A Survey](https://mlsanity.com/paper/2210.05952/) (2022)
- [Generative Adversarial Networks for Brain Images Synthesis: A Review](https://mlsanity.com/paper/2305.15421/) (2023)
- [SynCellFactory: Generative Data Augmentation for Cell Tracking](https://mlsanity.com/paper/2404.16421/) (2024)
- [Metrics to Quantify Global Consistency in Synthetic Medical Images](https://mlsanity.com/paper/2308.00402/) (2023)
- [Devil is in Details: Locality-Aware 3D Abdominal CT Volume Generation for Self-Supervised Organ Segmentation](https://mlsanity.com/paper/2409.20332/) (2024)
- [Synthesising Handwritten Music with GANs: A Comprehensive Evaluation of CycleWGAN, ProGAN, and DCGAN](https://mlsanity.com/paper/2411.16405/) (2024)
- [MedLoRD: A Medical Low-Resource Diffusion Model for High-Resolution 3D CT Image Synthesis](https://mlsanity.com/paper/2503.13211/) (2025)
- [Scalable next-scale autoregression for medical image generation across anatomical regions](https://mlsanity.com/paper/2602.14512/) (2026)

## Abstract (arXiv)

Generative Adversarial Networks (GAN) have many potential medical imaging applications, including data augmentation, domain adaptation, and model explanation. Due to the limited memory of Graphical Processing Units (GPUs), most current 3D GAN models are trained on low-resolution medical images, these models either cannot scale to high-resolution or are prone to patchy artifacts. In this work, we propose a novel end-to-end GAN architecture that can generate high-resolution 3D images. We achieve this goal by using different configurations between training and inference. During training, we adopt a hierarchical structure that simultaneously generates a low-resolution version of the image and a randomly selected sub-volume of the high-resolution image. The hierarchical design has two advantages: First, the memory demand for training on high-resolution images is amortized among sub-volumes. Furthermore, anchoring the high-resolution sub-volumes to a single low-resolution image ensures anatomical consistency between sub-volumes. During inference, our model can directly generate full high-resolution images. We also incorporate an encoder with a similar hierarchical structure into the model to extract features from the images. Experiments on 3D thorax CT and brain MRI demonstrate that our approach outperforms state of the art in image generation. We also demonstrate clinical applications of the proposed model in data augmentation and clinical-relevant feature extraction.