# MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images

arXiv: https://arxiv.org/abs/2310.03559 · HTML: https://mlsanity.com/paper/2310.03559/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 59 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation](https://mlsanity.com/paper/2309.06380/) (2023)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks](https://mlsanity.com/paper/1807.10225/) (2018)
- [Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization](https://mlsanity.com/paper/2104.05833/) (2021)
- [RoentGen: Vision-Language Foundation Model for Chest X-ray Generation](https://mlsanity.com/paper/2211.12737/) (2022)
- [Differentially Private Diffusion Models Generate Useful Synthetic Images](https://mlsanity.com/paper/2302.13861/) (2023)
- [Generative AI for Medical Imaging: extending the MONAI Framework](https://mlsanity.com/paper/2307.15208/) (2023)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [Generation of 3D Brain MRI Using Auto-Encoding Generative Adversarial Networks](https://mlsanity.com/paper/1908.02498/) (2019)
- [Hierarchical Amortized Training for Memory-efficient High Resolution 3D GAN](https://mlsanity.com/paper/2008.01910/) (2020)
- [SADM: Sequence-Aware Diffusion Model for Longitudinal Medical Image Generation](https://mlsanity.com/paper/2212.08228/) (2022)
- [Medical Diffusion: Denoising Diffusion Probabilistic Models for 3D Medical Image Generation](https://mlsanity.com/paper/2211.03364/) (2022)
- [3D Convolutional Encoder-Decoder Network for Low-Dose CT via Transfer Learning from a 2D Trained Network](https://mlsanity.com/paper/1802.05656/) (2018)

## Built on by (most cited first, 30 of 41)

- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [CLIP in Medical Imaging: A Survey](https://mlsanity.com/paper/2312.07353/) (2023)
- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [RadGPT: Constructing 3D Image-Text Tumor Datasets](https://mlsanity.com/paper/2501.04678/) (2025)
- [Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging](https://mlsanity.com/paper/2510.20639/) (2025)
- [Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model](https://mlsanity.com/paper/2505.04522/) (2025)
- [UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysis](https://mlsanity.com/paper/2510.15710/) (2025)
- [Taming Vision-Language Models for Medical Image Analysis: A Comprehensive Review](https://mlsanity.com/paper/2506.18378/) (2025)
- [TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency](https://mlsanity.com/paper/2507.00802/) (2025)
- [MedDiff-FM: A Diffusion-based Foundation Model for Versatile Medical Image Applications](https://mlsanity.com/paper/2410.15432/) (2024)
- [A Generalist Model for Diverse Text-Guided Medical Image Synthesis](https://mlsanity.com/paper/2405.09806/) (2024)
- [Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey](https://mlsanity.com/paper/2409.18214/) (2024)
- [From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation](https://mlsanity.com/paper/2506.00633/) (2025)
- [WASABI: A Metric for Evaluating Morphometric Plausibility of Synthetic Brain MRIs](https://mlsanity.com/paper/2504.21771/) (2025)
- [Prompt to Polyp: Medical Text-Conditioned Image Synthesis with Diffusion Models](https://mlsanity.com/paper/2505.05573/) (2025)
- [CTFlow: Video-Inspired Latent Flow Matching for 3D CT Synthesis](https://mlsanity.com/paper/2508.12900/) (2025)
- [Mammo-FM: Breast-specific foundational model for Integrated Mammographic Diagnosis, Prognosis, and Reporting](https://mlsanity.com/paper/2512.00198/) (2025)
- [MedLoRD: A Medical Low-Resource Diffusion Model for High-Resolution 3D CT Image Synthesis](https://mlsanity.com/paper/2503.13211/) (2025)
- [Radiology Report Conditional 3D CT Generation with Multi Encoder Latent diffusion Model](https://mlsanity.com/paper/2509.14780/) (2025)
- [ImmunoDiff: A Diffusion Model for Immunotherapy Response Prediction in Lung Cancer](https://mlsanity.com/paper/2505.23675/) (2025)
- [An Anisotropic Cross-View Texture Transfer with Multi-Reference Non-Local Attention for CT Slice Interpolation](https://mlsanity.com/paper/2509.20242/) (2025)
- [Unpaired Image-to-Image Translation via a Self-Supervised Semantic Bridge](https://mlsanity.com/paper/2602.16664/) (2026)
- [T2ID-CAS: Diffusion Model and Class Aware Sampling to Mitigate Class Imbalance in Neck Ultrasound Anatomical Landmark Detection](https://mlsanity.com/paper/2504.21231/) (2025)
- [Leveraging Clinical Text and Class Conditioning for 3D Prostate MRI Generation](https://mlsanity.com/paper/2506.10230/) (2025)
- [TauGenNet: Plasma-Driven Tau PET Image Synthesis via Text-Guided 3D Diffusion Models](https://mlsanity.com/paper/2509.04269/) (2025)
- [Integrating Anatomical Priors into a Causal Diffusion Model](https://mlsanity.com/paper/2509.09054/) (2025)
- [Cycle Diffusion Model for Counterfactual Image Generation](https://mlsanity.com/paper/2509.24267/) (2025)
- [MeDUET: Disentangled Unified Pretraining for 3D Medical Image Synthesis and Analysis](https://mlsanity.com/paper/2602.17901/) (2026)
- [Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation](https://mlsanity.com/paper/2603.08305/) (2026)
- [CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis](https://mlsanity.com/paper/2605.12650/) (2026)

## Abstract (arXiv)

This paper introduces an innovative methodology for producing high-quality 3D lung CT images guided by textual information. While diffusion-based generative models are increasingly used in medical imaging, current state-of-the-art approaches are limited to low-resolution outputs and underutilize radiology reports' abundant information. The radiology reports can enhance the generation process by providing additional guidance and offering fine-grained control over the synthesis of images. Nevertheless, expanding text-guided generation to high-resolution 3D images poses significant memory and anatomical detail-preserving challenges. Addressing the memory issue, we introduce a hierarchical scheme that uses a modified UNet architecture. We start by synthesizing low-resolution images conditioned on the text, serving as a foundation for subsequent generators for complete volumetric data. To ensure the anatomical plausibility of the generated samples, we provide further guidance by generating vascular, airway, and lobular segmentation masks in conjunction with the CT images. The model demonstrates the capability to use textual input and segmentation tasks to generate synthesized images. The results of comparative assessments indicate that our approach exhibits superior performance compared to the most advanced models based on GAN and diffusion techniques, especially in accurately retaining crucial anatomical features such as fissure lines, airways, and vascular structures. This innovation introduces novel possibilities. This study focuses on two main objectives: (1) the development of a method for creating images based on textual prompts and anatomical components, and (2) the capability to generate new images conditioning on anatomical elements. The advancements in image generation can be applied to enhance numerous downstream tasks.