# Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation

arXiv: https://arxiv.org/abs/2605.30893 · HTML: https://mlsanity.com/paper/2605.30893/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 23 of 23)

- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [CV-VAE: A Compatible Video VAE for Latent Generative Video Models](https://mlsanity.com/paper/2405.20279/) (2024)
- [Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes](https://mlsanity.com/paper/2002.04752/) (2020)
- [Towards Generalizable Tumor Synthesis](https://mlsanity.com/paper/2402.19470/) (2024)
- [GenerateCT: Text-Conditional Generation of 3D Chest CT Volumes](https://mlsanity.com/paper/2305.16037/) (2023)
- [VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging](https://mlsanity.com/paper/2406.05285/) (2024)
- [WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.17459/) (2024)
- [Label-Free Liver Tumor Segmentation](https://mlsanity.com/paper/2303.14869/) (2023)
- [Improved Video VAE for Latent Video Diffusion Model](https://mlsanity.com/paper/2411.06449/) (2024)
- [Are Natural Domain Foundation Models Useful for Medical Image Classification?](https://mlsanity.com/paper/2310.19522/) (2023)
- [MedVAE: Efficient Automated Interpretation of Medical Images with Large-Scale Generalizable Autoencoders](https://mlsanity.com/paper/2502.14753/) (2025)
- [Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imaging](https://mlsanity.com/paper/2510.20639/) (2025)
- [Large Motion Video Autoencoding with Cross-modal Video VAE](https://mlsanity.com/paper/2412.17805/) (2024)
- [LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models](https://mlsanity.com/paper/2503.14325/) (2025)
- [ReXGroundingCT: A 3D Chest CT Dataset for Segmentation of Findings from Free-Text Reports](https://mlsanity.com/paper/2507.22030/) (2025)
- [FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis](https://mlsanity.com/paper/2406.01264/) (2024)
- [Text-Driven Tumor Synthesis](https://mlsanity.com/paper/2412.18589/) (2024)
- [Text2CT: Towards 3D CT Volume Generation from Free-text Descriptions Using Diffusion Model](https://mlsanity.com/paper/2505.04522/) (2025)
- [Analyzing Tumors by Synthesis](https://mlsanity.com/paper/2409.06035/) (2024)
- [CT-SGAN: Computed Tomography Synthesis GAN](https://mlsanity.com/paper/2110.09288/) (2021)

## Built on by (most cited first, 1 of 1)

- [MRIComp4Flow: Compression of 3D Brain MRI for Training Multi-Modal Generative Models](https://mlsanity.com/paper/2608.10291/) (2026)

## Abstract (arXiv)

Variational autoencoders (VAEs) compress high resolution CT volumes into compact latents while preserving clinically relevant structure. However, training CT-specific VAEs from scratch or heavily fine-tuning them incurs substantial computational and engineering cost, and often degrades under heterogeneous scanners, protocols, and diseases. This paper makes a progressive stride toward training-free medical VAEs by leveraging a critical observation: a single Foundation VAE, pretrained at scale on natural images and videos, can serve as a unified interface for CT Reconstruction, Augmentation, and Generation. With both encoder and decoder frozen, the Foundation VAE reconstructs CT volumes with preserved anatomy while suppressing acquisition noise; training segmentation models on these reconstructions improves surface accuracy by 3.9% NSD on average for pancreatic tumor and lung tumor. Within the same Foundation VAE latent space, a conditional latent diffusion model achieves 3.9% lower average FVD with 36.2% higher CT CLIP score, and improves multi-disease generation faithfulness across 18 types by 2.76% AUC. These results demonstrate Foundation VAEs as a practical interface for scalable CT representation reuse and faithful CT generation. Our code and demo are available at https://github.com/qic999/Foundation-VAE.