# Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis

arXiv: https://arxiv.org/abs/2411.02372 · HTML: https://mlsanity.com/paper/2411.02372/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 113 references that are in the map.
- First posted on arXiv 4 Nov 2024.

## Builds on (most cited first, 30 of 71)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Contrastive Learning for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2007.15651/) (2020)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [TotalSegmentator: robust segmentation of 104 anatomical structures in CT images](https://mlsanity.com/paper/2208.05868/) (2022)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [MONAI: An open-source framework for deep learning in healthcare](https://mlsanity.com/paper/2211.02701/) (2022)
- [Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis](https://mlsanity.com/paper/2111.14791/) (2021)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation](https://mlsanity.com/paper/2206.08023/) (2022)
- [Understanding Diffusion Models: A Unified Perspective](https://mlsanity.com/paper/2208.11970/) (2022)
- [Contrastive learning of global and local features for medical image segmentation with limited annotations](https://mlsanity.com/paper/2006.10511/) (2020)
- [Med3D: Transfer Learning for 3D Medical Image Analysis](https://mlsanity.com/paper/1904.00625/) (2019)
- [nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation](https://mlsanity.com/paper/2404.09556/) (2024)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining](https://mlsanity.com/paper/2107.09559/) (2021)
- [CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection](https://mlsanity.com/paper/2301.00785/) (2023)
- [Generalist Foundation Models from a Multimodal Dataset for 3D Computed Tomography](https://mlsanity.com/paper/2403.17834/) (2024)
- [TorchIO: A Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning](https://mlsanity.com/paper/2003.04696/) (2020)
- [StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners](https://mlsanity.com/paper/2306.00984/) (2023)

## Built on by (most cited first, 29 of 29)

- [MultiMorph: On-demand Atlas Construction](https://mlsanity.com/paper/2504.00247/) (2025)
- [Does DINOv3 Set a New Medical Vision Standard? Benchmarking 2D and 3D Classification, Segmentation, and Registration](https://mlsanity.com/paper/2509.06467/) (2025)
- [Domain-randomized deep learning for neuroimage analysis](https://mlsanity.com/paper/2507.13458/) (2025)
- [Unraveling Normal Anatomy via Fluid-Driven Anomaly Randomization](https://mlsanity.com/paper/2501.13370/) (2025)
- [A 3D Cross-modal Keypoint Descriptor for MR-US Matching and Registration](https://mlsanity.com/paper/2507.18551/) (2025)
- [PolyPose: Deformable 2D/3D Registration via Polyrigid Transformations](https://mlsanity.com/paper/2505.19256/) (2025)
- [Robust Fetal Pose Estimation across Gestational Ages via Cross-Population Augmentation](https://mlsanity.com/paper/2509.12062/) (2025)
- [Cardiovascular disease classification using radiomics and geometric features from cardiac CT](https://mlsanity.com/paper/2506.22226/) (2025)
- [CardioComposer: Leveraging Differentiable Geometry for Compositional Control of Anatomical Diffusion Models](https://mlsanity.com/paper/2509.08015/) (2025)
- [A Scalable Distributed Framework for Multimodal GigaVoxel Image Registration](https://mlsanity.com/paper/2509.25044/) (2025)
- [Deep infant brain segmentation from multi-contrast MRI](https://mlsanity.com/paper/2512.05114/) (2025)
- [Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision](https://mlsanity.com/paper/2603.13660/) (2026)
- [Pixel-level Counterfactual Contrastive Learning for Medical Image Segmentation](https://mlsanity.com/paper/2603.17110/) (2026)
- [VoxCor: Training-Free Volumetric Features for Multimodal Voxel Correspondence](https://mlsanity.com/paper/2605.13798/) (2026)
- [How Far Has AI Come in Liver Fibrosis Staging? A Large-Scale Real-World Dataset and Benchmark](https://mlsanity.com/paper/2605.25595/) (2026)
- [Rapid patient-specific neural networks for intraoperative X-ray to volume registration](https://mlsanity.com/paper/2503.16309/) (2025)
- [Surrogate Supervision for Robust and Generalizable Deformable Image Registration](https://mlsanity.com/paper/2509.09869/) (2025)
- [Anatomica: Localized Control over Geometric and Topological Properties for Anatomical Diffusion Models](https://mlsanity.com/paper/2511.20587/) (2025)
- [Pancakes: Consistent Multi-Protocol Image Segmentation Across Biomedical Domains](https://mlsanity.com/paper/2512.13534/) (2025)
- [Free Lunch in Medical Image Foundation Model Pre-training via Randomized Synthesis and Disentanglement](https://mlsanity.com/paper/2602.12317/) (2026)
- [FDIF: Formula-Driven supervised Learning with Implicit Functions for 3D Medical Image Segmentation](https://mlsanity.com/paper/2603.23199/) (2026)
- [Why Invariance is Not Enough for Biomedical Domain Generalization and How to Fix It](https://mlsanity.com/paper/2604.02564/) (2026)
- [A Multimodal Clinically Informed Coarse-to-Fine Framework for Longitudinal CT Registration in Proton Therapy](https://mlsanity.com/paper/2604.13397/) (2026)
- [Patient-Agnostic Synthetic Pretraining for Efficient Patient-Specific Intraoperative 2D/3D Registration](https://mlsanity.com/paper/2607.23343/) (2026)
- [CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space](https://mlsanity.com/paper/2609.00272/) (2026)
- [The microscope is the mask: privileged views and labels from a cryo-ET forward model](https://mlsanity.com/paper/2609.04325/) (2026)
- [FleXray: Universal Clinical X-ray Segmentation](https://mlsanity.com/paper/2609.26756/) (2026)
- [Temporal Diffusion Bridge Model for Vessel Extraction Using Physics-informed Synthetic Data](https://mlsanity.com/paper/2610.14105/) (2026)
- [Registration is a Representation Problem: Solving Whole-Body PET/CT Registration](https://mlsanity.com/paper/2610.18804/) (2026)

## Abstract (arXiv)

Current volumetric biomedical foundation models struggle to generalize as public 3D datasets are small and do not cover the broad diversity of medical procedures, conditions, anatomical regions, and imaging protocols. We address this by creating a representation learning method that instead anticipates strong domain shifts at training time itself. We first propose a data engine that synthesizes highly variable training samples that would enable generalization to new biomedical contexts. To then train a single 3D network for any voxel-level task, we develop a contrastive learning method that pretrains the network to be stable against nuisance imaging variation simulated by the data engine, a key inductive bias for generalization. This network's features can be used as robust representations of input images for downstream tasks and its weights provide a strong, dataset-agnostic initialization for finetuning on new datasets. As a result, we set new standards across both multimodality registration and few-shot segmentation, a first for any 3D biomedical vision model, all without (pre-)training on any existing dataset of real images.