mlsanityData to 7 Oct 2026

When and Where to Trust the Teacher: Unifying On-Policy Distillation and GRPO through Entropy-Calibrated Credit Assignment

arXiv 2609.28385 ↗First posted 23 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) supervises mathematical reasoning through final-answer correctness, but provides little guidance on individual tokens. On-policy distillation (OPD) supplies dense feedback on student-generated responses, yet teacher preference need not reflect correctness. Recent hybrids combine OPD and verifier-derived advantages or reweight task credit using teacher ratios. However, teacher guidance enters after verifier-based group normalization, and token reweighting need not preserve the total task credit assigned to each response. We introduce Unified Entropy-Calibrated Credit Redistribution for GRPO (UECR-GRPO), which integrates verifier and teacher signals within a single GRPO-style update at both the response and token levels. Path-Utility Unification (PUU) combines verifier reward and a teacher-to-anchor path log-ratio in a single KL-regularized objective. Its on-policy implementation uses a length-normalized teacher score and combines both rewards before group normalization and PPO clipping, allowing teacher evidence to influence the response ranking. Entropy-Calibrated Redistribution (ECR) then uses the signed teacher--old-policy token gap to redistribute the verifier-derived component. Full-vocabulary teacher entropy attenuates uncertain guidance, while a response-wise zero-sum projection preserves the total task credit and its token-wise sign before clipping. Across five mathematical reasoning benchmarks, UECR-GRPO achieves average (Avg@12) accuracies of 17.21% and 65.09% with Qwen3-1.7B and Qwen3-4B students, respectively, exceeding the strongest baseline at each scale by 0.89 and 0.56 percentage points.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 24 references that are in the map. First posted on arXiv 23 Sep 2026.

Builds on

21 of 24 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards17 Jun
  2. 2025KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning2 Jun
  3. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  4. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2025Group Sequence Policy Optimizationcited by 788
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  6. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  7. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  8. 2026Self-Distilled RLVRcited by 109
  9. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  10. 2026StepOPSD: Step-Aware Online Preference Self-Distillation for Agent Reinforcement Learningcited by 14
  11. 2026SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Samplingcited by 12
  12. 2026Reward-Gated On-Policy Distillationcited by 11
4 more
  1. 2026ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Taskscited by 7
  2. 2026Distilled Reinforcement Learning for LLM Post-trainingcited by 7
  3. 2026Keep Policy Gradient in Charge: Sibling-Guided Credit Distillation for Long-Horizon Tool-Use Agentscited by 6
  4. 2026Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers
  2. 2026Reward-Aligned Reweighting for On-Policy Distillation
  3. 2026DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation
  4. 2026On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers
  5. 2026Beyond the Best Teacher: Expanding and Compressing the Reasoning Solution Manifold
  6. 2026Calibrating Teacher--Student Discrepancy for On-Policy Distillation
  7. 2026TeacherGRPO: Closing the Capacity Gap in Reasoning Distillation via Teacher Alignment
  8. 2026Your Teacher Can't Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation