mlsanityData to 7 Oct 2026

MAD-OPD: Breaking the Ceiling in On-Policy Distillation via Multi-Agent Debate

arXiv 2605.01347 ↗First posted 2 May 2026cs.CL cs.AI cs.LGcited by 12

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

On-policy distillation (OPD) trains a student on its own trajectories under token-level teacher supervision, but existing methods are capped by a single-teacher capability ceiling: when the teacher errs, the student inherits the error. OPD also remains largely unexplored in agentic tasks, where per-step errors compound across long trajectories and destabilize training. We propose MAD-OPD (Multi-Agent Debate-driven On-Policy Distillation), which breaks this ceiling by recasting the distillation teacher as a deliberative collective of teachers that debate over the student's on-policy state; the debate produces an emergent collective intelligence that supplies token-level supervision, with each teacher's contribution weighted by its post-debate confidence. To extend OPD to agentic tasks, we also introduce On-Policy Agentic Distillation (OPAD), which adds step-level sampling to stabilize training under multi-step error compounding. We additionally derive a task-adaptive divergence principle, selecting JSD (Jensen-Shannon divergence) for agentic stability and reverse KL (Kullback-Leibler) divergence for code generation, and verify it both theoretically and empirically. Across six teacher-student configurations (Qwen3 and Qwen3.5; 1.7B-14B students, 8B-32B teachers) and five agentic and code benchmarks, MAD-OPD ranks first across all six configurations; on the 14B+8B→4B setting it lifts the agentic average by +2.4% and the code average by +3.7% over the stronger single-teacher OPD.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 40 references that are in the map. First posted on arXiv 2 May 2026.

Builds on

36 of 40 references.

References · 36

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  6. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  9. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
24 more
  1. 2015Policy Distillationcited by 414
  2. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  3. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  4. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  5. 2026Reinforcement Learning via Self-Distillationcited by 285
  6. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipecited by 262
  7. 2024ToolACE: Winning the Points of LLM Function Callingcited by 204
  8. 2026On-Policy Context Distillation for Language Modelscited by 161
  9. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  10. 2026Entropy-Aware On-Policy Distillation of Language Modelscited by 151
  11. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  12. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  13. 2026Self-Distilled RLVRcited by 109
  14. 2026Privileged Information Distillation for Language Modelscited by 79
  15. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 75
  16. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 74
  17. 2026Stable On-Policy Distillation through Adaptive Target Reformulationcited by 47
  18. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 45
  19. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  20. 2025Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learningcited by 39
  21. 2023GKD: A General Knowledge Distillation Framework for Large-scale Pre-trained Language Modelcited by 15
  22. 2025Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancementcited by 10
  23. 2025SMAGDi: Socratic Multi Agent Interaction Graph Distillation for Efficient High Accuracy Reasoningcited by 3
  24. 2023AMTSS: An Adaptive Multi-Teacher Single-Student Knowledge Distillation Framework For Multilingual Language Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  2. 2026PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learningcited by 6
  3. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  4. 2026From Proprietary to Open-Source: Bridging the Distribution Gap via Multi-Agent Protocol Distillation in Agentic Searchcited by 4
  5. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillationcited by 3
  6. 2026When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillationcited by 2
  7. 2026Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillationcited by 2
  8. 2026MIRROR: Learning from the Other View for Multi-Modal Reasoningcited by 1
  9. 2026From Parameters to Behaviors: A Survey of Model Fusion for Large Language Modelscited by 1
  10. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  11. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agentscited by 0
  12. 2026ChemOPD: Multi-Teacher On-Policy Distillation for Multi-Task Chemical Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Stochastic Teacher Intervention for Agentic On-Policy Distillation
  2. 2026Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers
  3. 2026SAGE-OPD: Selective Agent-Guided Intervention for Multi-Turn On-Policy Distillation
  4. 2026Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation
  5. 2026The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes
  6. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  7. 2026Beyond Imitation: Filtering On-Policy Distillation by Reasoning Progress
  8. 2026Look Ahead Before You Distill: Future Trajectory Validation of Teacher Guidance for Agentic On-Policy Distillation