# MedSyn2: Flexible Control of 3D CT Generation via Text and Semantically-Defined Segmentation Prompts

arXiv: https://arxiv.org/abs/2606.00967 · HTML: https://mlsanity.com/paper/2606.00967/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 31 May 2026.

## Builds on (most cited first, 30 of 33)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free](https://mlsanity.com/paper/2505.06708/) (2025)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [Inf-Net: Automatic COVID-19 Lung Infection Segmentation from CT Images](https://mlsanity.com/paper/2004.14133/) (2020)
- [Open-Sora 2.0: Training a Commercial-Level Video Generation Model in 200k](https://mlsanity.com/paper/2503.09642/) (2025)
- [Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models](https://mlsanity.com/paper/2211.10655/) (2022)
- [Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis](https://mlsanity.com/paper/2305.18453/) (2023)
- [GenerateCT: Text-Conditional Generation of 3D Chest CT Volumes](https://mlsanity.com/paper/2305.16037/) (2023)
- [MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images](https://mlsanity.com/paper/2310.03559/) (2023)
- [3D MedDiffusion: A 3D Medical Latent Diffusion Model for Controllable and High-quality Medical Image Generation](https://mlsanity.com/paper/2412.13059/) (2024)
- [MedGen3D: A Deep Generative Framework for Paired 3D Image and Mask Generation](https://mlsanity.com/paper/2304.04106/) (2023)
- [Pillar-0: A New Frontier for Radiology Foundation Models](https://mlsanity.com/paper/2511.17803/) (2025)
- [LaVin-DiT: Large Vision Diffusion Transformer](https://mlsanity.com/paper/2411.11505/) (2024)
- [MAISI-v2: Accelerated 3D High-Resolution Medical Image Synthesis with Rectified Flow and Region-specific Contrastive Loss](https://mlsanity.com/paper/2508.05772/) (2025)
- [PRISM: High-Resolution & Precise Counterfactual Medical Image Generation using Language-guided Stable Diffusion](https://mlsanity.com/paper/2503.00196/) (2025)
- [Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model](https://mlsanity.com/paper/2505.04522/) (2025)
- [Semantic Image Synthesis for Abdominal CT](https://mlsanity.com/paper/2312.06453/) (2023)
- [TRACE: Temporally Reliable Anatomically-Conditioned 3D CT Generation with Enhanced Efficiency](https://mlsanity.com/paper/2507.00802/) (2025)
- [Lung-DDPM: Semantic Layout-guided Diffusion Models for Thoracic CT Image Synthesis](https://mlsanity.com/paper/2502.15204/) (2025)
- [From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation](https://mlsanity.com/paper/2506.00633/) (2025)
- [CTFlow: Video-Inspired Latent Flow Matching for 3D CT Synthesis](https://mlsanity.com/paper/2508.12900/) (2025)
- [LAND: Lung and Nodule Diffusion for 3D Chest CT Synthesis with Anatomical Guidance](https://mlsanity.com/paper/2510.18446/) (2025)

## Abstract (arXiv)

Generative models for volumetric medical images have found many applications in medical imaging, ranging from data augmentation to serving as priors for inverse problems. For these applications, generating high-resolution 3D images with strong controllability is essential but remains highly challenging. Existing approaches typically control generation either through radiology reports used as text prompts or through full image segmentation. While text-based prompting is flexible, it provides limited spatial control over the location, shape, and boundary of abnormalities. In contrast, segmentation-based methods receive precise spatial guidance but are restrictive in requiring full-organ annotations. In this work, we propose a flexible multimodal framework for controllable volumetric image generation that supports input from radiology reports and segmentation prompts (both optional). Our approach allows users to provide segmentation of a specific anatomy or abnormality without requiring full-organ annotations. The semantic meaning of the segmentation mask is specified through an accompanying text description, resulting in a highly flexible and scalable conditioning mechanism. We develop a memory-efficient architecture based on a modified diffusion transformer that jointly processes image and segmentation tokens. The model further incorporates gated attention to effectively attend to long radiology reports. Experiments demonstrate that our method achieves state-of-the-art perceptual and semantic scores (e.g., 24% relative improvement in mean FID), generates high-resolution anatomically consistent CT volumes, and improves data efficiency when used for data augmentation. Radiologists' evaluation further confirms strong alignment between generated and real medical images.