# Synthetic Augmentation with Large-scale Unconditional Pre-training

arXiv: https://arxiv.org/abs/2308.04020 · HTML: https://mlsanity.com/paper/2308.04020/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Deep Learning for Medical Image Analysis](https://mlsanity.com/paper/1708.08987/) (2017)
- [The Effectiveness of Data Augmentation in Image Classification using Deep Learning](https://mlsanity.com/paper/1712.04621/) (2017)
- [Brain Imaging Generation with Latent Diffusion Models](https://mlsanity.com/paper/2209.07162/) (2022)
- [GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification](https://mlsanity.com/paper/1803.01229/) (2018)
- [Generating Images with Perceptual Similarity Metrics based on Deep Networks](https://mlsanity.com/paper/1602.02644/) (2016)
- [A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images](https://mlsanity.com/paper/2209.13167/) (2022)
- [Selective Synthetic Augmentation with HistoGAN for Improved Histopathology Image Classification](https://mlsanity.com/paper/2111.06399/) (2021)
- [PathologyGAN: Learning deep representations of cancer tissue](https://mlsanity.com/paper/1907.02644/) (2019)
- [Synthetic Augmentation and Feature-based Filtering for Improved Cervical Histopathology Image Classification](https://mlsanity.com/paper/1907.10655/) (2019)
- [Synthetic Sample Selection via Reinforcement Learning](https://mlsanity.com/paper/2008.11331/) (2020)
- [Exploring Compositional Visual Generation with Latent Classifier Guidance](https://mlsanity.com/paper/2304.12536/) (2023)

## Built on by (most cited first, 18 of 18)

- [Deep Generative Models for 3D Medical Image Synthesis](https://mlsanity.com/paper/2410.17664/) (2024)
- [Masked Conditional Diffusion Model for Enhancing Deepfake Detection](https://mlsanity.com/paper/2402.00541/) (2024)
- [Generative Medical Segmentation](https://mlsanity.com/paper/2403.18198/) (2024)
- [Image Distillation for Safe Data Sharing in Histopathology](https://mlsanity.com/paper/2406.13536/) (2024)
- [Iterative Online Image Synthesis via Diffusion Model for Imbalanced Classification](https://mlsanity.com/paper/2403.08407/) (2024)
- [Generating Realistic Counterfactuals for Retinal Fundus and OCT Images using Diffusion Models](https://mlsanity.com/paper/2311.11629/) (2023)
- [Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology](https://mlsanity.com/paper/2312.09792/) (2023)
- [Frequency-Calibrated Membership Inference Attacks on Medical Image Diffusion Models](https://mlsanity.com/paper/2506.14919/) (2025)
- [Enabling PSO-Secure Synthetic Data Sharing Using Diversity-Aware Diffusion Models](https://mlsanity.com/paper/2506.17975/) (2025)
- [Two-stage Cytopathological Image Synthesis for Augmenting Cervical Abnormality Screening](https://mlsanity.com/paper/2402.14707/) (2024)
- [An Ordinal Diffusion Model for Generating Medical Images with Different Severity Levels](https://mlsanity.com/paper/2403.00452/) (2024)
- [DiNO-Diffusion. Scaling Medical Diffusion via Self-Supervised Pre-Training](https://mlsanity.com/paper/2407.11594/) (2024)
- [Mask-guided cross-image attention for zero-shot in-silico histopathologic image generation with a diffusion model](https://mlsanity.com/paper/2407.11664/) (2024)
- [Synthetic Augmentation for Anatomical Landmark Localization using DDPMs](https://mlsanity.com/paper/2410.12489/) (2024)
- [Prototype-Guided Diffusion for Digital Pathology: Achieving Foundation Model Performance with Minimal Clinical Data](https://mlsanity.com/paper/2504.12351/) (2025)
- [Phenotype-Guided Generative Model for High-Fidelity Cardiac MRI Synthesis: Advancing Pretraining and Clinical Applications](https://mlsanity.com/paper/2505.03426/) (2025)
- [SemiGDA: Generative Dual-distribution Alignment for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2604.23274/) (2026)
- [Structural MRI Synthesis for Alzheimer's Disease via Conditional Diffusion on Anatomical Masks](https://mlsanity.com/paper/2606.18354/) (2026)

## Abstract (arXiv)

Deep learning based medical image recognition systems often require a substantial amount of training data with expert annotations, which can be expensive and time-consuming to obtain. Recently, synthetic augmentation techniques have been proposed to mitigate the issue by generating realistic images conditioned on class labels. However, the effectiveness of these methods heavily depends on the representation capability of the trained generative model, which cannot be guaranteed without sufficient labeled training data. To further reduce the dependency on annotated data, we propose a synthetic augmentation method called HistoDiffusion, which can be pre-trained on large-scale unlabeled datasets and later applied to a small-scale labeled dataset for augmented training. In particular, we train a latent diffusion model (LDM) on diverse unlabeled datasets to learn common features and generate realistic images without conditional inputs. Then, we fine-tune the model with classifier guidance in latent space on an unseen labeled dataset so that the model can synthesize images of specific categories. Additionally, we adopt a selective mechanism to only add synthetic samples with high confidence of matching to target labels. We evaluate our proposed method by pre-training on three histopathology datasets and testing on a histopathology dataset of colorectal cancer (CRC) excluded from the pre-training datasets. With HistoDiffusion augmentation, the classification accuracy of a backbone classifier is remarkably improved by 6.4% using a small set of the original labels. Our code is available at https://github.com/karenyyy/HistoDiffAug.