mlsanityData to 7 Oct 2026

Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling

arXiv 2509.01649 ↗First posted 1 Sep 2025cs.LGcited by 7

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

In the past year, distillation has seen a renewed prominence in large language model (LLM) pretraining, exemplified by the Llama-3.2 and Gemma model families. While distillation has historically been shown to improve statistical modeling, its effects on new paradigms that are key to modern LLMs, such as test-time scaling and in-context learning, remain underexplored. In this work, we make three main contributions. First, we show that pretraining with distillation yields models that exhibit remarkably better test-time scaling. Second, we observe that this benefit comes with a trade-off: distillation impairs in-context learning capabilities, particularly the one modeled via induction heads. Third, to demystify these findings, we study distilled pretraining in a sandbox of a bigram model, which helps us isolate the common principal factor behind our observations. Finally, using these insights, we shed light on various design choices for pretraining that should help practitioners going forward.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 63 references that are in the map. First posted on arXiv 1 Sep 2025.

Builds on

44 of 63 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Distillation Scaling Laws12 Feb

Other references · 43

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Let's Verify Step by Stepcited by 3,598
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2025s1: Simple test-time scalingcited by 1,311
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  11. 2022In-context Learning and Induction Headscited by 930
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
31 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  3. 2019Improved Knowledge Distillation via Teacher Assistantcited by 567
  4. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  5. 2019On the Efficacy of Knowledge Distillationcited by 377
  6. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  7. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  8. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  9. 2024Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoningcited by 267
  10. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  11. 2015Unifying distillation and privileged informationcited by 218
  12. 2020Self-Distillation Amplifies Regularization in Hilbert Spacecited by 171
  13. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 160
  14. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  15. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  16. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 81
  17. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  18. 2025Multi-Agent Verification: Scaling Test-Time Compute with Multiple Verifierscited by 58
  19. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  20. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  21. 2021Dynamic Knowledge Distillation for Pre-trained Language Modelscited by 38
  22. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  23. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 28
  24. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  25. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  26. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  27. 2023On student-teacher deviations in distillation: does it pay to disobey?cited by 16
  28. 2025Multi-Token Prediction Needs Registerscited by 16
  29. 2024Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliancecited by 9
  30. 2025Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanationcited by 8
  31. 2025Looking beyond the next tokencited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026Ministral 3cited by 88
  2. 2025Pre-training under infinite computecited by 24
  3. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 5
  4. 2026Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memorycited by 4
  5. 2026Computational Arbitrage in AI Model Marketscited by 1
  6. 2026GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectificationcited by 1
  7. 2026Let the Data Decide: Supervision Analysis, Capability Trade-offs, and Adaptive Objective Routing in Continued Pre-Training via Off-Policy Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  2. 2026Curriculum Learning-Guided Progressive Distillation in Large Language Models
  3. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Data
  4. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
  5. 2025UNDO: Understanding Distillation as Optimization
  6. 2026Thinking at the Right Size: Amortized Distillation Across Post-Trained LLMs
  7. 2026Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models
  8. 2022Learning by Distilling Context