mlsanityData to 14 Oct 2026

Synthetic bootstrapped pretraining

arXiv 2509.15248 ↗First posted 17 Sep 2025cs.CL cs.AIcited by 6

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training. While the standard pretraining teaches LMs to learn causal correlations among tokens within a single document, it is not designed to efficiently model the rich, learnable inter-document correlations that can potentially lead to better performance. We validate SBP by designing a compute-matched pretraining setup and pretrain a 3B-parameter and a 6B-parameter model on up to 1T tokens from scratch. We find SBP consistently improves upon a strong repetition baseline and delivers up to 60% of performance improvement attainable by an oracle upper bound with access to 20x more unique data. Qualitative analysis reveals that the synthesized documents go beyond mere paraphrases -- SBP first abstracts a core concept from the seed material and then crafts a new narration on top of it. Besides strong empirical performance, SBP admits a natural Bayesian interpretation: the synthesizer implicitly learns to abstract the latent concepts shared between related documents.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 46 of its 62 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

46 of 62 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
34 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  8. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  12. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  13. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  14. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  15. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  16. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  17. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  18. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  19. 2023Scaling Data-Constrained Language Modelscited by 403
  20. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  21. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  22. 2019Large Memory Layers with Product Keyscited by 124
  23. 2017Generating Sentences by Editing Prototypescited by 88
  24. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  25. 2025Self-Adapting Language Modelscited by 84
  26. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  27. 2025Distillation Scaling Lawscited by 67
  28. 2024Synthetic continued pretrainingcited by 60
  29. 2025Reasoning to Learn from Latent Thoughtscited by 45
  30. 2024How predictable is language model benchmark performance?cited by 37
  31. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  32. 2022Decoupled Context Processing for Context Augmented Language Modelingcited by 24
  33. 2023ResMem: Learn what you can and memorize the restcited by 13
  34. 2024SOAR: Improved Indexing for Approximate Nearest Neighbor Searchcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Data-efficient pre-training by scaling synthetic megadocscited by 7
  2. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  3. 2026Synthetic Data for any Differentiable Targetcited by 2
  4. 2026Bridging Compute- and Data-Optimal Pretrainingcited by 2
  5. 2026Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAGcited by 1
  6. 2026AFCS: Atomic-Fact-Conditioned Synthesis for Factual Knowledge Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
  2. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
  3. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
  4. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
  5. 2025Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
  6. 2025Scaling Laws of Synthetic Data for Language Models
  7. 2023Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings
  8. 2026Generating Pretraining Tokens from Organic Data for Data-Bound Scaling