mlsanityData to 14 Oct 2026

Not All Synthetic Data Is Yours to Learn From

arXiv 2605.31126 ↗First posted 29 May 2026cs.CL cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 2 more
Abstract (arXiv)

Can a language model improve from plain text sampled from itself, with no prompts, no teacher, no verifier, and no reward model? Yes, but only when the synthetic corpus is compatible with the student, a relational property of the source-student pair rather than an intrinsic property of the data. We call this the latent capability resurfacing hypothesis: weak self-training can amplify capabilities already present in the pretrained model, but only under this compatibility condition. We study this in the minimal setting of prompt-free unconditional self-training, where base language models are fine-tuned on text generated from the BOS token alone, with no task specification or external supervision. We report three findings. First, synthetic utility is relational rather than intrinsic: self-generated data is the most effective source, same-lineage transfer outperforms stronger but differently trained sources, and cross-family transfer is substantially weaker. Second, common intrinsic proxies fail: neither benchmark-level semantic similarity nor average per-token likelihood under the student predicts which corpora help. Third, this regime produces a surprising byproduct. In controlled Pythia experiments, capability and verbatim memorization decouple: benchmark utility is preserved or improved while held-out exact-match extraction drops by over 95 percent, with no forget set, privacy objective, or targeted unlearning. Together, these results suggest that prompt-free self-training works by amplifying what the student already knows, not by importing structure from the data. They also reveal a regime in which capability and verbatim memorization can be separated without any explicit unlearning objective.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 33 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

30 of 33 references.

References · 30

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
18 more
  1. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  3. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 437
  4. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  5. 2023Self-Consuming Generative Models Go MADcited by 250
  6. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  7. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 89
  8. 2025Llama-Embed-Nemotron-8B: A Universal Text Embedding Model for Multilingual and Cross-Lingual Taskscited by 51
  9. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  10. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  11. 2024Self-Improving Diffusion Models with Synthetic Datacited by 35
  12. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 31
  13. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 20
  14. 2026Soft Contamination Means Benchmarks Test Shallow Generalizationcited by 8
  15. 2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
  16. 2025Neon: Negative Extrapolation From Self-Training Improves Image Generationcited by 5
  17. 2026From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Spacecited by 3
  18. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  2. 2025Does Training on Synthetic Data Make Models Less Robust?
  3. 2025Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modeling
  4. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  5. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
  6. 2024Self-Improvement in Language Models: The Sharpening Mechanism
  7. 2026Break Step: Recursive Training Resonates with Replayed Sampling Noise
  8. 2026Self-Training Doesn't Flatten Language -- It Restructures It: Surface Markers Amplify While Deep Syntax Dies