mlsanityData to 7 Oct 2026

Learning from Synthetic Data: Limitations of ERM

arXiv 2601.15468 ↗First posted 21 Jan 2026cs.LG cs.DS stat.MLcited by 3

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

The prevalence and low cost of LLMs have led to a rise of synthetic content. From review sites to court documents, "natural" content has been contaminated by data points that appear similar to natural data, but are in fact LLM-generated. In this work we revisit fundamental learning theory questions in this, now ubiquitous, setting. We model this scenario as a sequence of learning tasks where the input is a mix of natural and synthetic data, and the learning algorithms are oblivious to the origin of any individual example. We study the possibilities and limitations of ERM in this setting. For the problem of estimating the mean of an arbitrary d-dimensional distribution, we find that while ERM converges to the true mean, it is outperformed by an algorithm that assigns non-uniform weights to examples from different generations of data. For the PAC learning setting, the disparity is even more stark. We find that ERM does not always converge to the true concept, echoing the model collapse literature. However, we show there are algorithms capable of learning the correct hypothesis for arbitrary VC classes and arbitrary amounts of contamination.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 14 of its 20 references that are in the map. First posted on arXiv 21 Jan 2026.

Builds on

14 of 20 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Maximizing the Potential of Synthetic Data: Insights from Random Matrix Theory11 Oct

Other references · 13

  1. 2023Self-Consuming Generative Models Go MADcited by 239
  2. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  3. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  4. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 75
  5. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  6. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  7. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  8. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  9. 2024Universality of the π2/6 Pathway in Avoiding Model Collapsecited by 24
  10. 2024Rate of Model Collapse in Recursive Trainingcited by 22
  11. 2025Escaping Collapse: The Strength of Weak Data for Large Language Model Trainingcited by 11
  12. 2025When Models Don't Collapse: On the Consistency of Iterative MLEcited by 11
1 more
  1. 2025Learning from positive and unlabeled examples -Finite size sample boundscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuningcited by 1
  2. 2026Pushing the Limits of High-Resolution Weather Forecasting through Data Scalingcited by 0
  3. 2026Minimax bounds for watermarked and masked recursive discrete distribution estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Synthetic Data Characterization via Training Dynamics
  2. 2026Training-Aware Target Coverage for Synthetic Data Selection
  3. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold
  4. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models
  5. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  6. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  7. 2026Learning with Synthetic Data via SGD in High-Dimensional Linear Regression
  8. 2023On the Limitation of Diffusion Models for Synthesizing Training Datasets