mlsanityData to 14 Oct 2026

Break Step: Recursive Training Resonates with Replayed Sampling Noise

arXiv 2609.11149 ↗First posted 10 Sep 2026cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

How fast does a language model degrade when trained on its own outputs? Theory traces it to gradually accumulating errors, while experiments report repeated phrases within ten generations. Under a fixed sampling seed in vLLM, the fast loss of lexical diversity comes from the sampler. When vLLM serves a batch from one seeded sampling configuration, every request receives the same random draws, and a fixed seed replays them every generation. Fine-tuning raises the tokens that won, and the replayed draws let them win by more. Sharing across requests and replay across generations matter only together. Remove either one, by changing the shared seed every generation or by giving each request its own seed that repeats every generation, and the unique-4-gram fraction of two StableLM checkpoints stays near its starting value of about 0.98 through generation 3. Keep both, and the replayed shared seed takes seven checkpoints from five families to between 0.045 and 0.38 by then. Three generations of replay write the favoured phrases into the weights: decoded with one seed per request, the generation-3 weights of the replayed StableLM-2-1.6B chain recover most of their diversity, yet the phrase that filled every sample under the shared seed still opens 46% of them. Without replay, five checkpoints drift slowly, consistent with the gradual accumulation that theory describes, and three turn incoherent though their diversity scores stay high. One peer-reviewed model-collapse pipeline that fine-tunes Gemma-2-27B samples identical prompts under one seeded configuration, and three quarters of the rows it released for one iteration repeat nearly as often as one such batch copies them. A seed per request restores the fresh sample that stability analyses assume.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 51 references that are in the map. First posted on arXiv 10 Sep 2026.

Builds on

35 of 51 references.

References · 35

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  7. 2023A Watermark for Large Language Modelscited by 725
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  9. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  10. 2023Scaling Data-Constrained Language Modelscited by 403
  11. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  12. 2023Robust Distortion-free Watermarks for Language Modelscited by 320
23 more
  1. 2023Self-Consuming Generative Models Go MADcited by 250
  2. 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
  3. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  4. 2014A* Samplingcited by 151
  5. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 151
  6. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  7. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  8. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  9. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  10. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  11. 2024Stable LM 2 1.6B Technical Reportcited by 90
  12. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  13. 2023On the Learnability of Watermarks for Language Modelscited by 81
  14. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  15. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 79
  16. 2024Strong Model Collapsecited by 65
  17. 2024Watermarking Makes Language Models Radioactivecited by 58
  18. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  19. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  20. 2024Self-Correcting Self-Consuming Loops for Generative Model Trainingcited by 37
  21. 2024GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trickcited by 37
  22. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  23. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget
  2. 2026Language Generation with Replay: A Learning-Theoretic View of Model Collapse
  3. 2025A Probabilistic Perspective on Model Collapse
  4. 2026Experimental evidence of progressive ChatGPT models self-convergence
  5. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  6. 2025Machine-generated text detection prevents language model collapse
  7. 2026SynthSentry: Detecting Synthetic Data Contamination in Language Model Training Data
  8. 2025Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?