mlsanityData to 14 Oct 2026

Let the Data Decide: Supervision Analysis, Capability Trade-offs, and Adaptive Objective Routing in Continued Pre-Training via Off-Policy Distillation

arXiv 2607.16246 ↗First posted 26 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Off-policy distillation is now central to large language model pre-training, yet how training data, objective parameterization, and model capabilities interact remains poorly characterized. We studies top-k-truncated, temperature-scaled off-policy distillation by decomposing this problem into two questions: an objective-to-capability analysis of how the training objective shapes token-level supervision and downstream performance, and a data-to-objective analysis of how data heterogeneity should inform objective routing. We first show that the language-modeling objective (LLM) and the knowledge-distillation objective (LKD) induce systematically different capability profiles, and trace this divergence to a gradient-level tension between direct observed-token reinforcement and teacher-supported alternative supervision. To quantify this tension, we introduce diagnostic metrics -- support coverage, observed-token probability mass, and teacher-distribution concentration -- and show via controlled sweeps that the support size k governs a coverage-sharpness trade-off, while distillation temperature controls within-support probability allocation. We then examine adaptive objective routing: a domain-level policy that applies LLM to math and code and LKD to general-domain data yields consistent gains over both single-objective baselines, whereas token-level routing based on observed-token probability mass or teacher entropy fails to consistently match the single-objective baseline. These results suggest that effective objective routing depends less on routing granularity than on the quality of the routing signal, reframing continued pre-training via off-policy distillation as a structured, data-conditional supervision-design problem rather than a global hyperparameter choice.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 20 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

19 of 20 references.

References · 19

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  8. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 245
  9. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  10. 2025LFM2 Technical Reportcited by 37
  11. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27
  12. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
7 more
  1. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 22
  2. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 16
  3. 2025LLM-Oriented Token-Adaptive Knowledge Distillationcited by 13
  4. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  5. 2026Don't Ignore the Tail: Decoupling top-K Probabilities for Efficient Language Model Distillationcited by 7
  6. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  7. 2026EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfercited by 2

Built on by

Not cited yet.