mlsanityData to 14 Oct 2026

Trust-Region Behavior Blending for On-Policy Distillation

arXiv 2605.31159 ↗First posted 29 May 2026cs.LG cs.AIcited by 8

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

On-policy distillation (OPD) trains a student on prefixes sampled from its own policy while matching a stronger teacher. This addresses the prefix mismatch of offline distillation, but early student rollouts can still be poor, placing teacher supervision on weak or low-quality prefixes. We propose Trust-Region behavior Blending (TRB), a warmup method that replaces the early rollout policy with the closest-to-teacher behavior policy inside a student-centered KL trust region, while keeping the per-prefix reverse-KL OPD loss unchanged. The KL budget is annealed to zero, so training returns to pure student rollouts after warmup. Across two math-reasoning distillation settings, TRB attains the strongest average among the compared methods.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 16 of its 19 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

16 of 19 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026TIP: Token Importance in On-Policy Distillation15 Apr
  2. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe14 Apr
  3. 2026Entropy-Aware On-Policy Distillation of Language Models7 Mar
  4. 2026Stable On-Policy Distillation through Adaptive Target Reformulation12 Jan

Other references · 12

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  10. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  11. 2024Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Samplingcited by 90
  12. 2025MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistantscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026When EOS Tokens Disagree: Understanding Length Inflation in On-Policy Distillationcited by 5
  2. 2026Simple-OPD: Demystifying Warm-up for On-policy Distillationcited by 4
  3. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progresscited by 3
  4. 2026Geometric Self-Distillation for Reasoning Generalizationcited by 2
  5. 2026CLOOPD: Closing the Learner Loop in On-Policy Distillationcited by 0
  6. 2026iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMscited by 0
  7. 2026SAKI: Maximal-Coupling-Routed Teacher Supervision for On-Policy Distillationcited by 0
  8. 2026Demystifying On-Policy Distillation: Where Does the Teacher's Advantage Live?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
  2. 2026Reward-Gated On-Policy Distillation
  3. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes
  4. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
  5. 2026When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
  6. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillation
  7. 2026Fast and Effective On-policy Distillation from Reasoning Prefixes
  8. 2026On-Policy Distillation with Negative-Policy Rollouts