mlsanityData to 14 Oct 2026

No Model Required: Text Entropy Rate Filtering Mitigates Iterative Fine-Tuning Collapse

arXiv 2610.01493 ↗First posted 1 Oct 2026cs.CL cs.AI cs.ITcited by 0

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data also fits: LLM synthetic data generation

Abstract (arXiv)

Iterative fine-tuning on synthetic data causes model collapse: output diversity narrows as rare patterns are progressively lost, a signature most visible as phrase-level repetition. Existing mitigations either require model log-probabilities, an external oracle, or continued access to real human data. Here we develop a new approach grounded in mathematical information theory: the non-parametric Kontoyiannis entropy rate estimator hk, computed entirely from raw text via match-length statistics, with no model of any kind. We show that this is in fact a superior training-data filter on text-diversity metrics in a fully-synthetic, single-lineage fine-tuning setting. In a six-generation QLoRA collapse experiment on Llama-3.1-8B, logprob-based filtering (the most established model-access-requiring baseline) provides no significant text-diversity benefit on any metric (p > 0.23), whereas hk-filtering yields +42% unique trigrams, +30% vocabulary, and -19% repetition (all p < 0.001). We validate hk as a cross-domain entropy proxy (β = 0.924, R2 = 0.746) and collapse detector (ρ = +0.454, p < 0.0001) across 4~domains, 2~temperatures, 2~generator--scorer model pairs, and 1,520 generated documents. Our results demonstrate that information theoretic approaches to collapse mitigation are efficient, and suggest new approaches for maintaining multi-agent diversity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 26 of its 34 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

26 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence18 Oct
  2. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops26 Feb
  3. 2024Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models16 Oct
  4. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification11 Jun
  5. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data1 Apr

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  8. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  9. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  10. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  11. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  12. 2023Self-Consuming Generative Models Go MADcited by 250
9 more
  1. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  2. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  3. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  4. 2024How to Synthesize Text Data without Model Collapse?cited by 25
  5. 2026Diversity Collapse in Multi-Agent LLM Systems: Structural Coupling and Collective Failure in Open-Ended Idea Generationcited by 7
  6. 2017Information flow reveals prediction limits in online social activitycited by 4
  7. 2022Information flow estimation: a study of news on Twittercited by 2
  8. 2017The quoter model: a paradigmatic model of the social flow of written informationcited by 1
  9. 2020Complex contagion features without social reinforcement in a model of social information flowcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Experimental evidence of progressive ChatGPT models self-convergence
  2. 2025Machine-generated text detection prevents language model collapse
  3. 2026Diversity in Large Language Models under Supervised Fine-Tuning
  4. 2025Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning
  5. 2025BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation
  6. 2026Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics
  7. 2026Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data
  8. 2026Tackling the Inherent Difficulty of Noise Filtering in RAG