mlsanityData to 7 Oct 2026

Black-Box On-Policy Distillation of Large Language Models

arXiv 2511.10643 ↗First posted 13 Nov 2025cs.CL cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

Black-box distillation creates student large language models (LLMs) by learning from a proprietary teacher model's text outputs alone, without access to its internal logits or parameters. In this work, we introduce Generative Adversarial Distillation (GAD), which enables on-policy and black-box distillation. GAD frames the student LLM as a generator and trains a discriminator to distinguish its responses from the teacher LLM's, creating a minimax game. The discriminator acts as an on-policy reward model that co-evolves with the student, providing stable, adaptive feedback. Experimental results show that GAD consistently surpasses the commonly used sequence-level knowledge distillation. In particular, Qwen2.5-14B-Instruct (student) trained with GAD becomes comparable to its teacher, GPT-5-Chat, on the LMSYS-Chat automatic evaluation. The results establish GAD as a promising and effective paradigm for black-box LLM distillation.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 13 Nov 2025.

Builds on

30 of 40 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2025Qwen3 Technical Reportcited by 8,195
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
18 more
  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  3. 2025s1: Simple test-time scalingcited by 1,311
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2023Instruction Tuning with GPT-4cited by 676
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  10. 2025LIMO: Less is More for Reasoningcited by 488
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  12. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  13. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  14. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  15. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  16. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  17. 2020LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuningcited by 25
  18. 2025Learning Reasoning Rewards from Expert Demonstrations with Inverse Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 53.

53 papers.

Sort:

By area:

  1. 2026On-Policy Context Distillation for Language Modelscited by 161
  2. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  3. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  4. 2026Uni-OPD: Unifying On-Policy Distillation with a Dual-Perspective Recipecited by 43
  5. 2026RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillationcited by 33
  6. 2026Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationcited by 28
  7. 2026Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Modelscited by 24
  8. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  9. 2026OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoningcited by 14
  10. 2026Prefix Teach, Suffix Fade: Local Teachability Collapse in Strong-to-Weak On-Policy Distillationcited by 13
  11. 2026Multi-Turn On-Policy Distillation with Prefix Replaycited by 12
  12. 2026Rubric-based On-policy Distillationcited by 11
38 more
  1. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  2. 2026Pass the Baton: Trajectory-Relayed On-Policy Distillationcited by 10
  3. 2026CORD: Bridging the Audio-Text Reasoning Gap via Weighted On-policy Cross-modal Distillationcited by 9
  4. 2026OVD: On-policy Verbal Distillationcited by 9
  5. 2026Beyond GRPO and On-Policy Distillation: An Empirical Sparse-to-Dense Reward Principle for Language-Model Post-Trainingcited by 9
  6. 2026Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RLcited by 8
  7. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  8. 2026Stealing Reasoning Traces from Proprietary LLM APIscited by 8
  9. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoningcited by 7
  10. 2026SPOT: Sparse Probing and Outcome Calibration for On-Policy Distillationcited by 7
  11. 2026SODA: Semi On-Policy Black-Box Distillation for Large Language Modelscited by 6
  12. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 5
  13. 2026OmniOPD: Logit-Free On-Policy Distillation via Speculative Verificationcited by 5
  14. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  15. 2026boldsymbolf-OPD: Stabilizing Long-Horizon On-Policy Distillation with Freshness-Aware Controlcited by 4
  16. 2026Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Modelscited by 3
  17. 2026Surgical Post-Training: Proximal On-Policy Distillation for Reasoning with Knowledge Retentioncited by 2
  18. 2026On-Policy Adversarial Flow Distillation for Autoregressive Video Generationcited by 2
  19. 2026When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizoncited by 2
  20. 2026Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrationscited by 2
  21. 2026A Few Teacher Steps Go a Long Way: Cost-Efficient On-Policy Data Augmentation for Agent Post-Trainingcited by 2
  22. 2026Step-Level On-Policy Distillation: Interpolating Between On-Policy Distillation and Supervised Fine-Tuningcited by 2
  23. 2026FAIL: Flow Matching Adversarial Imitation Learning for Image Generationcited by 1
  24. 2026OPOD: On-Policy Omni Distillationcited by 1
  25. 2026Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understandingcited by 0
  26. 2026Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimizationcited by 0
  27. 2026PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learningcited by 0
  28. 2026The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoningcited by 0
  29. 2026OptiAgent: End-to-End Optimization Modeling via Multi-Agent Iterative Refinementcited by 0
  30. 2026AdaStop: Cost-Aware Early Stopping for DNN Test Selectioncited by 0
  31. 2026Proxy OPD: On-Policy Distillation with Transferable Relative Proxy Updatecited by 0
  32. 2026Aha-Flow Distillation: Flow Markers Matter in LLM Reasoningcited by 0
  33. 2026PSD: Pseudo Self-Distillation of Memory Representation Capabilities for LLM Agentscited by 0
  34. 2026Post-Training Leaves Behavioral Shadows on Unrelated Decisionscited by 0
  35. 2026Persistent Negatives for Adversarial Black-Box On-Policy Distillationcited by 0
  36. 2026USA: Update-aware SAM for Cross-domain On-Policy Disitllation of Language Agentscited by 0
  37. 2026Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillationcited by 0
  38. 2026The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillationcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Adversarial Moment-Matching Distillation of Large Language Models
  2. 2025Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
  3. 2025Knowledge Distillation with Training Wheels
  4. 2026Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe
  5. 2026A Survey of On-Policy Distillation for Large Language Models
  6. 2025Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
  7. 2026Estimating the Black-box LLM Uncertainty with Distribution-Aligned Adversarial Distillation
  8. 2026Train Ahead, Distill Back: Bootstrapping On-Policy Self-Distillation for Large Language Models