# Is Synthetic Data all We Need? Benchmarking the Robustness of Models Trained with Synthetic Images

arXiv: https://arxiv.org/abs/2405.20469 · HTML: https://mlsanity.com/paper/2405.20469/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 86 references that are in the map.
- First posted on arXiv 30 May 2024.

## Builds on (most cited first, 30 of 71)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)

## Built on by (most cited first, 30 of 41)

- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [SegEarth-OV: Towards Training-Free Open-Vocabulary Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2410.01768/) (2024)
- [SimMLM: A Simple Framework for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2507.19264/) (2025)
- [A CLIP-Powered Framework for Robust and Generalizable Data Selection](https://mlsanity.com/paper/2410.11215/) (2024)
- [GLASS: Guided Latent Slot Diffusion for Object-Centric Learning](https://mlsanity.com/paper/2407.17929/) (2024)
- [Detecting Human Artifacts from Text-to-Image Models](https://mlsanity.com/paper/2411.13842/) (2024)
- [ReME: A Data-Centric Framework for Training-Free Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.21233/) (2025)
- [SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection](https://mlsanity.com/paper/2411.05633/) (2024)
- [CC-Diff: Enhancing Contextual Coherence in Remote Sensing Image Synthesis](https://mlsanity.com/paper/2412.08464/) (2024)
- [Annotation-Free Open-Vocabulary Segmentation for Remote-Sensing Images](https://mlsanity.com/paper/2508.18067/) (2025)
- [A Little Human Data Goes A Long Way](https://mlsanity.com/paper/2410.13098/) (2024)
- [When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators](https://mlsanity.com/paper/2602.19946/) (2026)
- [DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scaling](https://mlsanity.com/paper/2409.16949/) (2024)
- [AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data](https://mlsanity.com/paper/2503.05665/) (2025)
- [Light-weight Fine-tuning Method for Defending Adversarial Noise in Pre-trained Medical Vision-Language Models](https://mlsanity.com/paper/2407.02716/) (2024)
- [Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classification](https://mlsanity.com/paper/2409.17091/) (2024)
- [Virus Infection Attack on LLMs: Your Poisoning Can Spread "VIA" Synthetic Data](https://mlsanity.com/paper/2509.23041/) (2025)
- [DIAGen: Semantically Diverse Image Augmentation with Generative Models for Few-Shot Learning](https://mlsanity.com/paper/2408.14584/) (2024)
- [Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines](https://mlsanity.com/paper/2511.14440/) (2025)
- [LoRA-Loop: Closing the Synthetic Replay Cycle for Continual VLM Learning](https://mlsanity.com/paper/2507.13568/) (2025)
- [How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Images](https://mlsanity.com/paper/2508.21565/) (2025)
- [Modularity is the Bedrock of Natural and Artificial Intelligence](https://mlsanity.com/paper/2602.18960/) (2026)
- [On the Difficulty of Learning a Meta-network for Training Data Selection](https://mlsanity.com/paper/2606.00571/) (2026)
- [Predictive uncertainty estimation in deep learning for lung carcinoma classification in digital pathology under real dataset shifts](https://mlsanity.com/paper/2408.08432/) (2024)
- [Will the Inclusion of Generated Data Amplify Bias Across Generations in Future Image Classification Models?](https://mlsanity.com/paper/2410.10160/) (2024)
- [Low-Biased General Annotated Dataset Generation](https://mlsanity.com/paper/2412.10831/) (2024)
- [Standardisation of Convex Ultrasound Data Through Geometric Analysis and Augmentation](https://mlsanity.com/paper/2502.09482/) (2025)
- [Exploring the Equivalence of Closed-Set Generative and Real Data Augmentation in Image Classification](https://mlsanity.com/paper/2508.09550/) (2025)
- [Detect Fake with Fake: Leveraging Synthetic Data-driven Representation for Synthetic Image Detection](https://mlsanity.com/paper/2409.08884/) (2024)
- [A Mutual Information Perspective on Multiple Latent Variable Generative Models for Positive View Generation](https://mlsanity.com/paper/2501.13718/) (2025)

## Abstract (arXiv)

A long-standing challenge in developing machine learning approaches has been the lack of high-quality labeled data. Recently, models trained with purely synthetic data, here termed synthetic clones, generated using large-scale pre-trained diffusion models have shown promising results in overcoming this annotation bottleneck. As these synthetic clone models progress, they are likely to be deployed in challenging real-world settings, yet their suitability remains understudied. Our work addresses this gap by providing the first benchmark for three classes of synthetic clone models, namely supervised, self-supervised, and multi-modal ones, across a range of robustness measures. We show that existing synthetic self-supervised and multi-modal clones are comparable to or outperform state-of-the-art real-image baselines for a range of robustness metrics - shape bias, background bias, calibration, etc. However, we also find that synthetic clones are much more susceptible to adversarial and real-world noise than models trained with real data. To address this, we find that combining both real and synthetic data further increases the robustness, and that the choice of prompt used for generating synthetic images plays an important part in the robustness of synthetic clones.