mlsanityData to 7 Oct 2026

DuoOPD: Learning from Joint Teacher-Student Outcomes for Multi-Task On-Policy Distillation

arXiv 2609.33711 ↗First posted 27 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) trains a student on its own responses with token-level feedback from a stronger teacher, yet the teacher can fail on questions the student already answers correctly, and how often each model succeeds varies across tasks. OPD ignores these outcomes and, on average, pushes down even the student's correct responses; gating feedback by student correctness fixes the direction but uses the teacher in the same way whether or not it succeeded. We introduce DuoOPD, in which the student's outcome sets the direction of feedback and the joint teacher-student outcome decides how the teacher supports it: when only the teacher succeeds, its verified answer becomes context for scoring the student's failed response, and when only the student succeeds, a weight shared within the task reinforces the whole response. A single rule covers all four outcome combinations without task-specific settings. Across Qwen3 and Llama, DuoOPD outperforms all five baselines in mean macro accuracy, improving over OPD by 2.58 and 5.98 percentage points, and it also leads on two further task mixtures spanning scientific calculation, instruction following, and code generation. Ablations show that outcome-based direction alone stays near the gated baseline, while the joint-outcome designs supply most of the gain.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

29 of 31 references.

References · 29

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  12. 2015Unifying distillation and privileged informationcited by 218
17 more
  1. 2026On-Policy Context Distillation for Language Modelscited by 161
  2. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  3. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  4. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  5. 2026MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Trainingcited by 60
  6. 2019Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understandingcited by 37
  7. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixescited by 37
  8. 2025Agent4Edu: Generating Learner Response Data by Generative Agents for Intelligent Education Systemscited by 32
  9. 2026Filter, Then Reweight: Rethinking Optimization Granularity in On-Policy Distillationcited by 31
  10. 2026Rethinking On-Policy Self-Distillation for Thinking Modelscited by 28
  11. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 12
  12. 2026On-policy Distillation with Verifiable Rewardcited by 10
  13. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  14. 2026H2SD: Hybrid Hindsight Self-Distillationcited by 5
  15. 2026One Student, Many Teachers: Multi-Task On-Policy Distillation via Soft-Prompt Privileged Contextcited by 4
  16. 2026Verify Before You Distill: Prompt-Level Teacher Gating for On-Policy Distillationcited by 4
  17. 2026Edu-Theater: A Data-Efficient Agent Framework for Scalable Learner Behavior Simulation through Staging Roll-Callcited by 2

Built on by

Not cited yet.