mlsanityData to 7 Oct 2026

Synthetic continued pretraining

arXiv 2409.07431 ↗First posted 11 Sep 2024cs.LG cs.AI cs.CLcited by 56

Not on the map. Why?

Abstract (arXiv)

Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-inefficient--to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it. This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities. Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them. If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation. To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Builds on 74 of its 105 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

74 of 105 references.

References · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  11. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  12. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2016Progressive Neural Networkscited by 1,332
  13. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  14. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  15. 2017Continual Learning with Deep Generative Replaycited by 1,152
  16. 2022Mass-Editing Memory in a Transformercited by 976
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  18. 2020Efficient Transformers: A Surveycited by 784
  19. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  20. 2023Instruction Tuning with GPT-4cited by 676
  21. 2022Large Language Models Can Self-Improvecited by 656
  22. 2024Self-Rewarding Language Modelscited by 654
  23. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  24. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  25. 2021Fast Model Editing at Scalecited by 491
  26. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  27. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  28. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  29. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  30. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  31. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  32. 2023Scaling Data-Constrained Language Modelscited by 391
  33. 2023Llemma: An Open Language Model For Mathematicscited by 385
  34. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  35. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  36. 2017Variational Continual Learningcited by 335
  37. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  38. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 32; 2026: 19.

56 papers.

Sort:

By area:

  1. 2024Scaling Laws for Precisioncited by 92
  2. 2025Self-Adapting Language Modelscited by 71
  3. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  4. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  5. 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoningcited by 45
  6. 2024MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Codecited by 39
  7. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 39
  8. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
  9. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  10. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  11. 2025Llama-3.1-FoundationAI-SecurityLLM-Base-8B Technical Reportcited by 28
  12. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
38 more
  1. 2025Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfallscited by 24
  2. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  3. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  4. 2025Memorization vs. Reasoning: Updating LLMs with New Knowledgecited by 12
  5. 2025Learning Facts at Scale with Active Readingcited by 11
  6. 2026Understanding LoRA as Knowledge Memory: An Empirical Analysiscited by 11
  7. 2025Enhancing LLM-based Quantum Code Generation with Multi-Agent Optimization and Quantum Error Correctioncited by 10
  8. 2025Midtraining Bridges Pretraining and Posttraining Distributionscited by 10
  9. 2024Continual Memorization of Factoids in Language Modelscited by 8
  10. 2025EnronQA: Towards Personalized RAG over Private Documentscited by 8
  11. 2026FutureSim: Replaying World Events to Evaluate Adaptive Agentscited by 7
  12. 2025Generative Data Refinement: Just Ask for Better Datacited by 6
  13. 2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
  14. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Datacited by 6
  15. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  16. 2025Synthetic bootstrapped pretrainingcited by 4
  17. 2025ADEPT: Continual Pretraining via Adaptive Expansion and Dynamic Decoupled Tuningcited by 4
  18. 2025Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Modelscited by 3
  19. 2025Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalizationcited by 3
  20. 2026ReasonCACHE: Teaching LLMs To Reason Without Weight Updatescited by 3
  21. 2025Comparing Knowledge Injection Methods for LLMs in a Low-Resource Regimecited by 2
  22. 2025From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learningcited by 2
  23. 2026Synthetic Data for any Differentiable Targetcited by 2
  24. 2025Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Modelscited by 1
  25. 2025Bilinear representation mitigates reversal curse and enables consistent model editingcited by 1
  26. 2025Fine-Tuned Thoughts: Leveraging Chain-of-Thought Reasoning for Industrial Asset Health Monitoringcited by 1
  27. 2025ASVRI-Legal: Fine-Tuning LLMs with Retrieval Augmented Generation for Enhanced Legal Regulationcited by 1
  28. 2026DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Datacited by 1
  29. 2026Self-Play Pretraining with Zero Datacited by 1
  30. 2025Spatial Knowledge Graph-Guided Multimodal Synthesiscited by 0
  31. 2025Balancing Synthetic Data and Replay for Enhancing Task-Specific Capabilitiescited by 0
  32. 2025Prior Knowledge Makes It Possible: From Sublinear Graph Algorithms to LLM Test-Time Methodscited by 0
  33. 2026FineInstructions: Scaling Synthetic Instructions to Pre-Training Scalecited by 0
  34. 2026Agentic Adversarial QA for Improving Domain-Specific LLMscited by 0
  35. 2026SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injectioncited by 0
  36. 2026Improving Latent Generalization Using Test-time Computecited by 0
  37. 2026ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?cited by 0
  38. 2026Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QAcited by 0

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Synthetic Mixed Training: Scaling Parametric Knowledge Acquisition Beyond RAG
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Training
  3. 2024Synthetic Knowledge Ingestion: Towards Knowledge Refinement and Injection for Enhancing Large Language Models
  4. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  5. 2026Time-Incremental Continued Pretraining of LLMs: Knowledge Updates Without Catastrophic Forgetting
  6. 2026Continually self-improving AI
  7. 2026Procedural Pretraining: Warming Up Language Models with Abstract Data
  8. 2024A synthetic data approach for domain generalization of NLI models