mlsanityData to 7 Oct 2026

How to Synthesize Text Data without Model Collapse?

arXiv 2412.14689 ↗First posted 19 Dec 2024cs.CL cs.AI cs.LGcited by 24

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Model collapse in synthetic data indicates that iterative training on self-generated data leads to a gradual decline in performance. With the proliferation of AI models, synthetic data will fundamentally reshape the web data ecosystem. Future GPT-n models will inevitably be trained on a blend of synthetic and human-produced data. In this paper, we focus on two questions: what is the impact of synthetic data on language model training, and how to synthesize data without model collapse? We first pre-train language models across different proportions of synthetic data, revealing a negative correlation between the proportion of synthetic data and model performance. We further conduct statistical analysis on synthetic data to uncover distributional shift phenomenon and over-concentration of n-gram features. Inspired by the above findings, we propose token editing on human-produced data to obtain semi-synthetic data. As a proof of concept, we theoretically demonstrate that token-level editing can prevent model collapse, as the test error is constrained by a finite upper bound. We conduct extensive experiments on pre-training from scratch, continual pre-training, and supervised fine-tuning. The results validate our theoretical proof that token-level editing improves model performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 53 of its 65 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

53 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Best Practices and Lessons Learned on Synthetic Data11 Apr

Other references · 52

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2024Mixtral of Expertscited by 1,785
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  9. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2023Textbooks Are All You Needcited by 561
  12. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
38 more
  1. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  2. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  4. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  5. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  6. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  7. 2024Bench2Drive: Towards Multi-Ability Benchmarking of Closed-Loop End-To-End Autonomous Drivingcited by 250
  8. 2023Self-Consuming Generative Models Go MADcited by 239
  9. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  10. 2024A Survey on Data Selection for Language Modelscited by 230
  11. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
  12. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  13. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  14. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  15. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  16. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  17. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  18. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 98
  19. 2024Stable LM 2 1.6B Technical Reportcited by 86
  20. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  21. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  22. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  23. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 75
  24. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  25. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  26. 2024Strong Model Collapsecited by 62
  27. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  28. 2024Think2Drive: Efficient Reinforcement Learning by Thinking in Latent World Model for Quasi-Realistic Autonomous Driving (in CARLA-v2)cited by 51
  29. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  30. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 43
  31. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  32. 2024Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talkcited by 34
  33. 2024Critical Data Size of Language Models from a Grokking Perspectivecited by 22
  34. 2023LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarioscited by 20
  35. 2024RAT: Adversarial Attacks on Deep Reinforcement Agents for Targeted Behaviorscited by 16
  36. 2023Normalization Enhances Generalization in Visual Reinforcement Learningcited by 13
  37. 2024On Domain-Adaptive Post-Training for Multimodal Large Language Modelscited by 13
  38. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 14.

24 papers.

Sort:

By area:

  1. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  2. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  3. 2025Machine-generated text detection prevents language model collapsecited by 14
  4. 2025Reformulation for Pretraining Data Augmentationcited by 9
  5. 2025Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Taskscited by 9
  6. 2025Hermes 4 Technical Reportcited by 5
  7. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
  8. 2025Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Themcited by 4
  9. 2025A Survey of LLM × DATAcited by 4
  10. 2026Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillationcited by 2
  11. 2026When AI Reviews Its Own Code: Recursive Self-Training Collapse in Code LLMscited by 2
  12. 2025What Matters in LLM-generated Data: Diversity and Its Effect on Model Fine-Tuningcited by 1
12 more
  1. 2026Large Language Models are Shannon Lossy Compressors Not Solomonoff Induction Estimators: Self-improvement and Singularity Are Not Near Without Symbolic Model Synthesiscited by 1
  2. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1
  3. 2026Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuningcited by 1
  4. 2026SIGMA: Scalable Spectral Insights for LLM Model Collapsecited by 0
  5. 2026Large Language Models Can Take False First Steps at Inference-time Planningcited by 0
  6. 2026The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Datacited by 0
  7. 2026Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Trainingcited by 0
  8. 2026Recursive LLM Degradation in Biomedical Question Answering: A Cross-Generation Studycited by 0
  9. 2026Comparison of techniques for fine-tuning open-weight models for entity extraction from radiology reportscited by 0
  10. 2026Effective Synthetic Data Curation Requires Group-Level Signalscited by 0
  11. 2026Feature Selective Model Collapse in Diffusion Models: Total Replacement versus Fixed-Budget Trainingcited by 0
  12. 2026No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapsecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
  2. 2025Scaling Laws of Synthetic Data for Language Models
  3. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  4. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  5. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  6. 2025BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
  7. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  8. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations