# MedVAE: Efficient Automated Interpretation of Medical Images with Large-Scale Generalizable Autoencoders

arXiv: https://arxiv.org/abs/2502.14753 · HTML: https://mlsanity.com/paper/2502.14753/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Builds on (most cited first, 22 of 22)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Contrastive Learning of Medical Visual Representations from Paired Images and Text](https://mlsanity.com/paper/2010.00747/) (2020)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [Domain Adaptation for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2102.09508/) (2021)
- [Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation](https://mlsanity.com/paper/2010.10042/) (2020)
- [Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset](https://mlsanity.com/paper/2406.06512/) (2024)
- [Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains](https://mlsanity.com/paper/2210.04133/) (2022)
- [VinDr-Mammo: A large-scale benchmark dataset for computer-aided diagnosis in full-field digital mammography](https://mlsanity.com/paper/2203.11205/) (2022)
- [Development and Validation of Deep Learning Algorithms for Detection of Critical Findings in Head CT Scans](https://mlsanity.com/paper/1803.05854/) (2018)
- [Exploring Image Augmentations for Siamese Representation Learning with Chest X-Rays](https://mlsanity.com/paper/2301.12636/) (2023)
- [Adapting Pre-trained Vision Transformers from 2D to 3D through Weight Inflation Improves Medical Image Segmentation](https://mlsanity.com/paper/2302.04303/) (2023)
- [CSAW-M: An Ordinal Classification Dataset for Benchmarking Mammographic Masking of Cancer](https://mlsanity.com/paper/2112.01330/) (2021)
- [Integrating AI into Radiology workflow: Levels of research, production, and feedback maturity](https://mlsanity.com/paper/1910.06424/) (2019)
- [High performance on-demand de-identification of a petabyte-scale medical imaging data lake](https://mlsanity.com/paper/2008.01827/) (2020)
- [The EMory BrEast imaging Dataset (EMBED): A Racially Diverse, Granular Dataset of 3.5M Screening and Diagnostic Mammograms](https://mlsanity.com/paper/2202.04073/) (2022)

## Built on by (most cited first, 22 of 22)

- [Unpaired Image-to-Image Translation via a Self-Supervised Semantic Bridge](https://mlsanity.com/paper/2602.16664/) (2026)
- [Project Imaging-X: A Survey of 1000+ Open-Access Medical Imaging Datasets for Foundation Model Development](https://mlsanity.com/paper/2603.27460/) (2026)
- [Generative Drifting for Conditional Medical Image Generation](https://mlsanity.com/paper/2604.19736/) (2026)
- [Efficiency vs. Efficacy: Assessing the Compression Ratio-Dice Score Relationship through a Simple Benchmarking Framework for Cerebrovascular 3D Segmentation](https://mlsanity.com/paper/2510.03769/) (2025)
- [Attend what matters: Leveraging vision foundational models for breast cancer classification using mammograms](https://mlsanity.com/paper/2604.19350/) (2026)
- [CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging](https://mlsanity.com/paper/2604.22989/) (2026)
- [Sparse Representation Learning for Vessels](https://mlsanity.com/paper/2605.01382/) (2026)
- [The Learnability Gap in Medical Latent Diffusion](https://mlsanity.com/paper/2605.17087/) (2026)
- [Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation](https://mlsanity.com/paper/2605.30893/) (2026)
- [Tokenizer-Generator Coupling in Medical Image Generation](https://mlsanity.com/paper/2608.07713/) (2026)
- [OsteoFlow: Lyapunov-Guided Flow Distillation for Predicting Bone Remodeling after Mandibular Reconstruction](https://mlsanity.com/paper/2603.22421/) (2026)
- [HUydra: Full-Range Lung CT Synthesis via Multiple HU Interval Generative Modelling](https://mlsanity.com/paper/2603.23041/) (2026)
- [Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI](https://mlsanity.com/paper/2604.05171/) (2026)
- [Domain-Specific Latent Representations Improve the Fidelity of Diffusion-Based Medical Image Super-Resolution](https://mlsanity.com/paper/2604.12152/) (2026)
- [NeuroGAN-3D: Enhancing Intrinsic Functional Brain Networks via High-Fidelity 3D Generative Super-Resolution](https://mlsanity.com/paper/2605.08373/) (2026)
- [DiffSegLung: Diffusion Radiomic Distillation for Unsupervised Lung Pathology Segmentation](https://mlsanity.com/paper/2605.11758/) (2026)
- [Predictive Enhancement Calibration for Latent Breast MRI Virtual Contrast Enhancement](https://mlsanity.com/paper/2608.03612/) (2026)
- [A radiographic world model for clinical reasoning and evidence generation](https://mlsanity.com/paper/2609.07719/) (2026)
- [Anatomy-Decomposed Chest Computed Tomography (CT) Projections as Scalable Supervision for Bone Suppression in Chest Radiographs](https://mlsanity.com/paper/2609.24937/) (2026)
- [Complementary Roles of Radiomics and Foundation Representations in Renal Cell Carcinoma Classification: A Comparative Study of 2D and 3D CT Encodings](https://mlsanity.com/paper/2609.26463/) (2026)
- [Radiomics--Foundation Fusion for Interpretable RCC Classification: Internal Benchmarking and Exploratory External Transfer](https://mlsanity.com/paper/2609.26578/) (2026)
- [Beyond Discrimination: Evaluating Probability Calibration in Radiomics-Image Fusion for Renal Cell Carcinoma Classification](https://mlsanity.com/paper/2610.14151/) (2026)

## Abstract (arXiv)

Medical images are acquired at high resolutions with large fields of view in order to capture fine-grained features necessary for clinical decision-making. Consequently, training deep learning models on medical images can incur large computational costs. In this work, we address the challenge of downsizing medical images in order to improve downstream computational efficiency while preserving clinically-relevant features. We introduce MedVAE, a family of six large-scale 2D and 3D autoencoders capable of encoding medical images as downsized latent representations and decoding latent representations back to high-resolution images. We train MedVAE autoencoders using a novel two-stage training approach with 1,052,730 medical images. Across diverse tasks obtained from 20 medical image datasets, we demonstrate that (1) utilizing MedVAE latent representations in place of high-resolution images when training downstream models can lead to efficiency benefits (up to 70x improvement in throughput) while simultaneously preserving clinically-relevant features and (2) MedVAE can decode latent representations back to high-resolution images with high fidelity. Our work demonstrates that large-scale, generalizable autoencoders can help address critical efficiency challenges in the medical domain. Our code is available at https://github.com/StanfordMIMI/MedVAE.