# Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data

arXiv: https://arxiv.org/abs/2510.03264 · HTML: https://mlsanity.com/paper/2510.03264/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 23 of 23)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning](https://mlsanity.com/paper/2308.08747/) (2023)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models](https://mlsanity.com/paper/2504.11468/) (2025)
- [AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset](https://mlsanity.com/paper/2504.16891/) (2025)
- [OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling](https://mlsanity.com/paper/2506.20512/) (2025)
- [NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model](https://mlsanity.com/paper/2508.14444/) (2025)
- [Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning](https://mlsanity.com/paper/2507.00432/) (2025)
- [Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models](https://mlsanity.com/paper/2504.03624/) (2025)
- [Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuning](https://mlsanity.com/paper/2402.04833/) (2024)
- [AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy](https://mlsanity.com/paper/2506.13284/) (2025)
- [MegaMath: Pushing the Limits of Open Math Corpora](https://mlsanity.com/paper/2504.02807/) (2025)
- [Rethinking Reflection in Pre-Training](https://mlsanity.com/paper/2504.04022/) (2025)
- [Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning](https://mlsanity.com/paper/2504.13941/) (2025)
- [OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery](https://mlsanity.com/paper/2503.17604/) (2025)
- [The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs](https://mlsanity.com/paper/2507.07562/) (2025)
- [Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models](https://mlsanity.com/paper/2502.08130/) (2025)
- [Aligning Instruction Tuning with Pre-training](https://mlsanity.com/paper/2501.09368/) (2025)

## Built on by (most cited first, 27 of 27)

- [Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning](https://mlsanity.com/paper/2512.20848/) (2025)
- [On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models](https://mlsanity.com/paper/2512.07783/) (2025)
- [A Survey on LLM Mid-Training](https://mlsanity.com/paper/2510.23081/) (2025)
- [The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data](https://mlsanity.com/paper/2603.16177/) (2026)
- [On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training](https://mlsanity.com/paper/2601.07389/) (2026)
- [LegalOne: A Family of Foundation Models for Reliable Legal Reasoning](https://mlsanity.com/paper/2602.00642/) (2026)
- [Self-Improvement of Large Language Models: A Technical Overview and Future Outlook](https://mlsanity.com/paper/2603.25681/) (2026)
- [Understanding Reasoning from Pretraining to Post-Training](https://mlsanity.com/paper/2607.16097/) (2026)
- [ZAYA1-8B Technical Report](https://mlsanity.com/paper/2605.05365/) (2026)
- [MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources](https://mlsanity.com/paper/2509.25531/) (2025)
- [Data-Centric Lessons To Improve Speech-Language Pretraining](https://mlsanity.com/paper/2510.20860/) (2025)
- [Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Models](https://mlsanity.com/paper/2603.01293/) (2026)
- [Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training](https://mlsanity.com/paper/2603.02208/) (2026)
- [When Can LLMs Learn to Reason with Weak Supervision?](https://mlsanity.com/paper/2604.18574/) (2026)
- [DataComp-VLM: Improved Open Datasets for Vision-Language Models](https://mlsanity.com/paper/2606.28551/) (2026)
- [Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models](https://mlsanity.com/paper/2607.12463/) (2026)
- [Distributional Clarity: The Hidden Driver of RL-Friendliness in Large Language Models](https://mlsanity.com/paper/2601.06911/) (2026)
- [The Master Key Hypothesis: Unlocking Cross-Model Capability Transfer via Linear Subspace Alignment](https://mlsanity.com/paper/2604.06377/) (2026)
- [Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages](https://mlsanity.com/paper/2605.20285/) (2026)
- [Reasoning Quality Emerges Early: Data Curation for Reasoning Models](https://mlsanity.com/paper/2606.26797/) (2026)
- [Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning](https://mlsanity.com/paper/2511.10381/) (2025)
- [daVinci-LLM:Towards the Science of Pretraining](https://mlsanity.com/paper/2603.27164/) (2026)
- [Similar Models Learn Differently: Final-Window Pretraining Shapes Post-Training Beyond SFT](https://mlsanity.com/paper/2607.25063/) (2026)
- [Reasoning Core: Designing Broad Procedural Data for Completion-Supervised Reasoning Training](https://mlsanity.com/paper/2608.05148/) (2026)
- [SEA-LION-v4.8: A Technical Report](https://mlsanity.com/paper/2609.18310/) (2026)
- [Selecting Diverse SFT Traces Improves Post-RL Generalization](https://mlsanity.com/paper/2609.33780/) (2026)
- [Lasting Effects of Abstract Pretraining Beyond Perplexity](https://mlsanity.com/paper/2609.38764/) (2026)

## Abstract (arXiv)

The prevailing paradigm for enhancing the reasoning abilities of LLMs revolves around post-training on high-quality, reasoning-intensive data. While emerging literature suggests that reasoning data is increasingly incorporated also during the mid-training stage-a practice that is relatively more proprietary and less openly characterized-the role of such data in pretraining remains unclear. In particular, due to the opaqueness of pretraining corpora in most frontier models, the effect of reasoning data introduced at different phases of pre- and/or post-training is relatively less reported in the scientific literature. This raises several important questions: Is adding reasoning data earlier during pretraining any better than introducing it during post-training? Could earlier inclusion risk overfitting and harm generalization, or instead establish durable foundations that later fine-tuning cannot recover? We conduct the first systematic study of how reasoning data-varying in scale, diversity, and quality-affects LLM performance when introduced at different stages of training. We find that front-loading reasoning data into pretraining is critical (19% avg gain), establishing foundational capabilities that cannot be fully replicated by later-stage SFT, even with more data. We uncover an asymmetric principle for optimal data allocation: pretraining benefits most from broad diversity in reasoning patterns (11% avg gain), while SFT is more sensitive to data quality (15% avg gain). We show that high-quality pretraining data has latent effects, activated only after SFT, and that naively scaling SFT data can be detrimental, washing away the benefits of early reasoning injection. Our results challenge the conventional separation of language modeling and reasoning, providing a principled guide for strategically allocating data across the entire training pipeline to build more capable models.