mlsanityData to 14 Oct 2026

Do We Need All the Synthetic Data? Targeted Image Augmentation via Diffusion Models

arXiv 2505.21574 ↗First posted 27 May 2025cs.CV cs.LGcited by 2

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

Synthetically augmenting training datasets with diffusion models has become an effective strategy for improving the generalization of image classifiers. However, existing approaches typically increase dataset size by 10-30x and struggle to ensure generation diversity, leading to substantial computational overhead. In this work, we introduce TADA (TArgeted Diffusion Augmentation), a principled framework that selectively augments examples that are not learned early in training using faithful synthetic images that preserve semantic features while varying noise. We show that augmenting only this targeted subset consistently outperforms augmenting the entire dataset. Through theoretical analysis on a two-layer CNN, we prove that TADA improves generalization by promoting homogeneity in feature learning speed without amplifying noise. Extensive experiments demonstrate that by augmenting only 30-40% of the training data, TADA improves generalization by up to 2.8% across diverse architectures including ResNet, ViT, ConvNeXt, and Swin Transformer on CIFAR-10/100, TinyImageNet, and ImageNet, using optimizers such as SGD and SAM. Notably, TADA combined with SGD outperforms the state-of-the-art optimizer SAM on CIFAR-100 and TinyImageNet. Furthermore, TADA shows promising improvements on object detection benchmarks, demonstrating its applicability beyond image classification. Our code is available at https://github.com/BigML-CS-UCLA/TADA.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 55 of its 62 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

55 of 62 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Context-guided Responsible Data Augmentation with Diffusion Models12 Mar
  2. 2024GenMix: Effective Data Augmentation with Generative Diffusion Model Image Editing3 Dec
  3. 2024Inversion Circle Interpolation: Diffusion-based Image Augmentation for Data-scarce Classification29 Aug
  4. 2024Dataset Enhancement with Instance-Level Augmentations12 Jun
  5. 2024DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models5 Apr
  6. 2024Enhance Image Classification via Inter-Class Image Mixup with Diffusion Model28 Mar
  7. 2024DreamDA: Generative Data Augmentation with Diffusion Models19 Mar
  8. 2023Training on Thin Air: Improve Image Classification with Generated Data24 May
  9. 2023Synthetic Data from Diffusion Models Improves ImageNet Classification17 Apr
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  11. 2023Effective Data Augmentation With Diffusion Models7 Feb
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
2 more
  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 41

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Generative Adversarial Networkscited by 5,627
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  10. 2022A ConvNet for the 2020scited by 3,579
  11. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
29 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  4. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  5. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  6. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  7. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  8. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  9. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  10. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 227
  11. 2022DeepCore: A Comprehensive Library for Coreset Selection in Deep Learningcited by 147
  12. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  13. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  14. 2023Sharpness-Aware Gradient Matching for Domain Generalizationcited by 102
  15. 2023Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentationcited by 100
  16. 2022Benign Overfitting in Two-layer Convolutional Neural Networkscited by 91
  17. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  18. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  19. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  20. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  21. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  22. 2022Towards understanding how momentum improves generalization in deep learningcited by 40
  23. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  24. 2023Benign Overfitting for Two-layer ReLU Convolutional Neural Networkscited by 34
  25. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 28
  26. 2022Boomerang: Local sampling on image manifolds using diffusion modelscited by 16
  27. 2024Advancing Fine-Grained Classification by Structure and Subject Preserving Augmentationcited by 12
  28. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  29. 2024Changing the Training Data Distribution to Reduce Simplicity Bias Improves In-distribution Generalizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Active Flow Expansion for Out-of-Distribution Discovery: from Theory to Moleculescited by 3
  2. 2026Learning-State-Aware Dynamic Generative Data Augmentation on Small-Scale Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Advances in Diffusion Models for Image Data Augmentation: A Review of Methods, Models, Evaluation Metrics and Future Research Directions
  2. 2024Semantic Augmentation in Images using Language
  3. 2024Data Augmentation for Image Classification using Generative AI
  4. 2023Image retrieval outperforms diffusion models on data augmentation
  5. 2024Learning on Less: Constraining Pre-trained Model Learning for Generalizable Diffusion-Generated Image Detection
  6. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clones
  7. 2024Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning
  8. 2023Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation