# Retrieval-Augmented Anatomical Guidance for Text-to-CT Generation

arXiv: https://arxiv.org/abs/2603.08305 · HTML: https://mlsanity.com/paper/2603.08305/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 19 of its 30 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- Retrieval-augmented generation / [Medical and biomedical RAG](https://mlsanity.com/rag/medical-and-biomedical-rag/)

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [Diffusion Models for Medical Image Analysis: A Comprehensive Survey](https://mlsanity.com/paper/2211.07804/) (2022)
- [Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2010.00710/) (2020)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learning](https://mlsanity.com/paper/2103.02148/) (2021)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [GenerateCT: Text-Conditional Generation of 3D Chest CT Volumes](https://mlsanity.com/paper/2305.16037/) (2023)
- [MedSyn: Text-guided Anatomy-aware Synthesis of High-Fidelity 3D CT Images](https://mlsanity.com/paper/2310.03559/) (2023)
- [From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation](https://mlsanity.com/paper/2506.00633/) (2025)
- [XGeM: A Multi-Prompt Foundation Model for Multimodal Medical Data Generation](https://mlsanity.com/paper/2501.04614/) (2025)
- [Any-to-Any Vision-Language Model for Multimodal X-ray Imaging and Radiological Report Generation](https://mlsanity.com/paper/2505.01091/) (2025)
- [Radiology Report Conditional 3D CT Generation with Multi Encoder Latent diffusion Model](https://mlsanity.com/paper/2509.14780/) (2025)

## Built on by (most cited first, 1 of 1)

- [Compositional Cross-Modality Translation via Whole-Volume Multitask Latent Flow Matching](https://mlsanity.com/paper/2608.08135/) (2026)

## Abstract (arXiv)

Text-conditioned generative models for volumetric medical imaging provide semantic control but lack explicit anatomical guidance, often resulting in outputs that are spatially ambiguous or anatomically inconsistent. In contrast, structure-driven methods ensure strong anatomical consistency but typically assume access to ground-truth annotations, which are unavailable when the target image is to be synthesized. We propose a retrieval-augmented approach for Text-to-CT generation that integrates semantic and anatomical information under a realistic inference setting. Given a radiology report, our method retrieves a semantically related clinical case using a 3D vision-language encoder and leverages its associated anatomical annotation as a structural proxy. This proxy is injected into a text-conditioned latent diffusion model via a ControlNet branch, providing coarse anatomical guidance while maintaining semantic flexibility. Experiments on the CT-RATE dataset show that retrieval-augmented generation improves image fidelity and clinical consistency compared to text-only baselines, while additionally enabling explicit spatial controllability, a capability inherently absent in such approaches. Further analysis highlights the importance of retrieval quality, with semantically aligned proxies yielding consistent gains across all evaluation axes. This work introduces a principled and scalable mechanism to bridge semantic conditioning and anatomical plausibility in volumetric medical image synthesis. Code is available at https://github.com/arco-group/RAGText2CT.