mlsanityData to 14 Oct 2026

Synthetic Data Characterization via Training Dynamics

arXiv 2609.39447 ↗First posted 30 Sep 2026cs.CL cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation also fits: Model collapse from synthetic data

Abstract (arXiv)

Interpreting properties of LLM-generated data is important for understanding its utility and limitations across learning tasks. In this work, we characterize synthetic data through sample-level learnability, studying variation among LLM families and scales, alongside human-written data as a reference. We first generate synthetic datasets spanning single- and multi-label classification, labeling, and tree prediction tasks. We then derive empirical data distributions from encoder training dynamics for both machine and organic data, and estimate the robustness of these distributions across encoders. Finally, we evaluate how data selection strategies based on these learnability signals affect both data sources differently.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 50 of its 84 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

50 of 84 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls2 Oct
  2. 2024Evaluating Language Models as Synthetic Data Generators4 Dec

Other references · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  12. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
36 more
  1. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  2. 2023Whose Opinions Do Language Models Reflect?cited by 773
  3. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
  4. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  5. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  6. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  7. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
  8. 2022Large Language Models Are Reasoning Teacherscited by 329
  9. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  10. 2023Self-Consuming Generative Models Go MADcited by 250
  11. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  12. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  13. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  14. 2021How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Modelscited by 196
  15. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 189
  16. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  17. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  18. 2019Soft-Label Dataset Distillation and Text Dataset Distillationcited by 107
  19. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  20. 2020Training Question Answering Models From Synthetic Datacited by 95
  21. 2023Contrasting Linguistic Patterns in Human and LLM-Generated News Textcited by 62
  22. 2021Deep Active Learning by Leveraging Training Dynamicscited by 32
  23. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  24. 2024Prompt Public Large Language Models to Synthesize Data for Private On-device Applicationscited by 17
  25. 2024The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detectioncited by 15
  26. 2025Machine-generated text detection prevents language model collapsecited by 14
  27. 2025Comparing LLM-generated and human-authored news text using formal syntactic theorycited by 11
  28. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  29. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  30. 2025Synthetic Text Generation for Training Large Language Models via Gradient Matchingcited by 10
  31. 2025SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domainscited by 7
  32. 2025Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimationcited by 4
  33. 2025Increasing LLM Coding Capabilities through Diverse Synthetic Coding Taskscited by 3
  34. 2026More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMscited by 3
  35. 2023FTFT: Efficient and Robust Fine-Tuning by Transferring Training Dynamicscited by 2
  36. 2025Better Benchmarking LLMs for Zero-Shot Dependency Parsingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Effective Synthetic Data Curation Requires Group-Level Signals
  2. 2026Training-Aware Target Coverage for Synthetic Data Selection
  3. 2025Data Value in the Age of Scaling: Understanding LLM Scaling Dynamics Under Real-Synthetic Data Mixtures
  4. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  5. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Models
  6. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  7. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  8. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models