mlsanityData to 7 Oct 2026

Beyond Real Data: Synthetic Data through the Lens of Regularization

arXiv 2510.08095 ↗First posted 9 Oct 2025stat.ML cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Synthetic data can improve generalization when real data is scarce, but excessive reliance may introduce distributional mismatches that degrade performance. In this paper, we present a learning-theoretic framework to quantify the trade-off between synthetic and real data. Our approach leverages algorithmic stability to derive generalization error bounds, characterizing the optimal synthetic-to-real data ratio that minimizes expected test error as a function of the Wasserstein distance between the real and synthetic distributions. We motivate our framework in the setting of kernel ridge regression with mixed data, offering a detailed analysis that may be of independent interest. Our theory predicts the existence of an optimal ratio, leading to a U-shaped behavior of test error with respect to the proportion of synthetic data. Empirically, we validate this prediction on CIFAR-10 and a clinical brain MRI dataset. Our theory extends to the important scenario of domain adaptation, showing that carefully blending synthetic target data with limited source data can mitigate domain shift and enhance generalization. We conclude with practical guidance for applying our results to both in-domain and out-of-domain scenarios.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 82 of its 133 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

82 of 133 references.

References · 82

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  7. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  8. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  9. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  12. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
38 more
  1. 2021Diffusion Schr"odinger Bridge with Applications to Score-Based Generative Modelingcited by 594
  2. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  3. 2019Benign Overfitting in Linear Regressioncited by 544
  4. 2018A Survey of Unsupervised Deep Domain Adaptationcited by 361
  5. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  6. 2018Adversarially Robust Generalization Requires More Datacited by 318
  7. 2016Learning from Simulated and Unsupervised Images through Adversarial Trainingcited by 287
  8. 2023Effective Data Augmentation With Diffusion Modelscited by 282
  9. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  10. 2018GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classificationcited by 242
  11. 2023Self-Consuming Generative Models Go MADcited by 239
  12. 2021Does Knowledge Distillation Really Work?cited by 187
  13. 2018To understand deep learning we need to understand kernel learningcited by 186
  14. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  15. 2022Fake it till you make it: Learning transferable representations from synthetic ImageNet clonescited by 171
  16. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  17. 2019A Primer on PAC-Bayesian Learningcited by 130
  18. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  19. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  20. 2018GAN Augmentation: Augmenting Training Data using Generative Adversarial Networkscited by 107
  21. 2017Stability and Generalization of Learning Algorithms that Converge to Global Optimacited by 100
  22. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  23. 2019High probability generalization bounds for uniformly stable algorithms with nearly optimal ratecited by 97
  24. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  25. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
  26. 2020Transfer Learning from Synthetic to Real-Noise Denoising with Adaptive Instance Normalizationcited by 92
  27. 2019Sharper bounds for uniformly stable algorithmscited by 91
  28. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  29. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  30. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  31. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  32. 2020End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systemscited by 65
  33. 2024Strong Model Collapsecited by 62
  34. 2025Distillation Scaling Lawscited by 59
  35. 2020A Close Look at Deep Learning with Small Datacited by 54
  36. 2025Scaling Laws for Optimal Data Mixturescited by 49
  37. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  38. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  2. 2026When Sample Selection Bias Precipitates Model Collapsecited by 0