mlsanityData to 14 Oct 2026

Self-Improving Pretraining: using post-trained models to pretrain better models

arXiv 2601.21343 ↗First posted 29 Jan 2026cs.CL cs.AI cs.LGcited by 6

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large language models are classically trained in stages: pretraining on raw text followed by post-training for instruction following and reasoning. However, this separation creates a fundamental limitation: many desirable behaviors such as safety, factuality, overall generation quality, and reasoning ability are only added at a late stage, even though the patterns learned earlier strongly shape a model's capabilities. To tackle this issue, we introduce a new way to pretrain and mid-train models that incorporates these behaviors earlier. We utilize an existing strong, post-trained model to both rewrite pretraining data and to judge policy model rollouts, thus using reinforcement earlier in training. In our experiments, we show this can give strong gains in quality, safety, factuality and reasoning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 53 of its 63 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

53 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models2 Jan

Other references · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  6. 2018Deep contextualized word representationscited by 2,290
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  11. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  13. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  14. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  15. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  16. 2023Pretraining Language Models with Human Preferencescited by 247
  17. 2023Fine-tuning Language Models for Factualitycited by 221
  18. 2015Semi-supervised Sequence Learningcited by 198
  19. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  20. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  21. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  22. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 80
  23. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  24. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  25. 2025Reinforcement Pre-Trainingcited by 40
  26. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasingcited by 39
  27. 2025Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Modelscited by 31
  28. 2025Bridging Offline and Online Reinforcement Learning for LLMscited by 25
  29. 2025RLP: Reinforcement as a Pretraining Objectivecited by 25
  30. 2025Learning to Reason for Factualitycited by 24
  31. 2025Rewriting Pre-Training Data Boosts LLM Performance in Math and Codecited by 21
  32. 2025Reinforcement Learning on Pre-Training Datacited by 12
  33. 2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 11
  34. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 11
  35. 2024CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoningcited by 10
  36. 2025Reformulation for Pretraining Data Augmentationcited by 10
  37. 2025Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoningcited by 6
  38. 2025Planted in Pretraining, Swayed by Finetuning: A Case Study on the Origins of Cognitive Biases in LLMscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Recursive self-improvement of AI research agentscited by 11
  2. 2026Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stagescited by 1
  3. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflectioncited by 1
  4. 2026ExpRL: Exploratory RL for LLM Mid-Trainingcited by 0
  5. 2026Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuningcited by 0
  6. 2026Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recallcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning
  2. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  3. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
  4. 2025Towards a Unified View of Large Language Model Post-Training
  5. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Models
  6. 2026Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models
  7. 2026Mid-Training with Self-Generated Data Improves Reinforcement Learning in Language Models
  8. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models