mlsanityData to 14 Oct 2026

Effective Synthetic Data Curation Requires Group-Level Signals

arXiv 2610.00779 ↗First posted 30 Sep 2026cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data also fits: LLM synthetic data generation

Abstract (arXiv)

Synthetic data now is essential to LLM training, used to strengthen advanced capabilities such as autonomous and long-horizon task execution. Yet recent work shows that training on it at scale can degrade model generation, making it important to decide what synthetic data is worth training on. While current data curation practices do so with individual-level signals (i.e., estimates of each data sample's training utility in isolation), across pre-training and post-training settings we show that this is insufficient for synthetic data, and that group-level signals (i.e., estimates of utility that account for interactions among data samples) are necessary for effective data curation. First, we show that individual-level signals are blind to how samples jointly affect training: synthetic datasets with different compositions can be indistinguishable under individual-level influence yet differ sharply under group-level influence, and curating by the latter yields better downstream performance, particularly in generative capability. Second, we find that group-level signals matter more as training pipelines become increasingly synthetic: among widely used data curation methods, only those incorporating them improve over baseline, with gains increasing when weights capturing relations among samples are amplified. Finally, we translate these findings into practice -- for model developers under a compute budget, we offer a cheap diagnostic that prioritizes which groups of synthetic data most need group-level estimation, recovering much of the benefit of full group-level scoring at a fraction of the compute cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 59 of its 70 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

59 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Strong Model Collapse7 Oct

Other references · 58

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
38 more
  1. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  2. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  3. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  4. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  5. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  6. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  7. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  8. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  9. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  10. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  11. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  12. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  13. 2019Towards Efficient Data Valuation Based on the Shapley Valuecited by 281
  14. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  15. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
  16. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
  17. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  18. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  19. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  20. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 147
  21. 2024Nemotron-4 340B Technical Reportcited by 144
  22. 2022Data Banzhaf: A Robust Data Valuation Framework for Machine Learningcited by 132
  23. 2023DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Modelscited by 123
  24. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  25. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  26. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  27. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 94
  28. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  29. 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 79
  30. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  31. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  32. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
  33. 2021An Empirical Comparison of Instance Attribution Methods for NLPcited by 35
  34. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  35. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  36. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  37. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 26
  38. 2024How to Synthesize Text Data without Model Collapse?cited by 25

50 most cited of 58 All 58 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Synthetic Data Characterization via Training Dynamics
  2. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
  3. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
  4. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  5. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Models
  6. 2026Let Training Guide Selection: Online Synthetic Data Filtering via Real-Anchored Utility
  7. 2026Training-Aware Target Coverage for Synthetic Data Selection
  8. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation