# Dual Policy Distillation

arXiv: https://arxiv.org/abs/2006.04061 · HTML: https://mlsanity.com/paper/2006.04061/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2020.

## Built on by (most cited first, 25 of 25)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning](https://mlsanity.com/paper/2106.06135/) (2021)
- [Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimization](https://mlsanity.com/paper/2206.13033/) (2022)
- [Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models](https://mlsanity.com/paper/2604.08527/) (2026)
- [Differential Privacy Meets Federated Learning under Communication Constraints](https://mlsanity.com/paper/2101.12240/) (2021)
- [From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcare](https://mlsanity.com/paper/2409.09727/) (2024)
- [Differentially private Riemannian optimization](https://mlsanity.com/paper/2205.09494/) (2022)
- [Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners](https://mlsanity.com/paper/2401.00583/) (2023)
- [Banded Square Root Matrix Factorization for Differentially Private Model Training](https://mlsanity.com/paper/2405.13763/) (2024)
- [Learning Disentangled Representations for Time Series](https://mlsanity.com/paper/2105.08179/) (2021)
- [Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Training](https://mlsanity.com/paper/2306.08173/) (2023)
- [KnowRU: Knowledge Reusing via Knowledge Distillation in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2103.14891/) (2021)
- [Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks](https://mlsanity.com/paper/2111.12064/) (2021)
- [Dynamic Privacy Budget Allocation Improves Data Efficiency of Differentially Private Gradient Descent](https://mlsanity.com/paper/2101.07413/) (2021)
- [Bounding the Excess Risk for Linear Models Trained on Marginal-Preserving, Differentially-Private, Synthetic Data](https://mlsanity.com/paper/2402.04375/) (2024)
- [KnowSR: Knowledge Sharing among Homogeneous Agents in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2105.11611/) (2021)
- [Online Policy Distillation with Decision-Attention](https://mlsanity.com/paper/2406.05488/) (2024)
- [Meta-DiffuB: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration](https://mlsanity.com/paper/2410.13201/) (2024)
- [Progressive-Resolution Policy Distillation: Leveraging Coarse-Resolution Simulations for Time-Efficient Fine-Resolution Policy Learning](https://mlsanity.com/paper/2412.07477/) (2024)
- [Towards Personalized Preprocessing Pipeline Search](https://mlsanity.com/paper/2302.14329/) (2023)
- [Enhancing DP-SGD through Non-monotonous Adaptive Scaling Gradient Weight](https://mlsanity.com/paper/2411.03059/) (2024)
- [DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning](https://mlsanity.com/paper/2109.07380/) (2021)
- [Private Blind Model Averaging - Distributed, Non-interactive, and Convergent](https://mlsanity.com/paper/2211.02003/) (2022)
- [MiranDa: Mimicking the Learning Processes of Human Doctors to Achieve Causal Inference for Medication Recommendation](https://mlsanity.com/paper/2408.01445/) (2024)
- [Value Function Initialization for Knowledge Transfer and Jump-start in Deep Reinforcement Learning](https://mlsanity.com/paper/2508.09277/) (2025)

## Abstract (arXiv)

Policy distillation, which transfers a teacher policy to a student policy has achieved great success in challenging tasks of deep reinforcement learning. This teacher-student framework requires a well-trained teacher model which is computationally expensive. Moreover, the performance of the student model could be limited by the teacher model if the teacher model is not optimal. In the light of collaborative learning, we study the feasibility of involving joint intellectual efforts from diverse perspectives of student models. In this work, we introduce dual policy distillation(DPD), a student-student framework in which two learners operate on the same environment to explore different perspectives of the environment and extract knowledge from each other to enhance their learning. The key challenge in developing this dual learning framework is to identify the beneficial knowledge from the peer learner for contemporary learning-based reinforcement learning algorithms, since it is unclear whether the knowledge distilled from an imperfect and noisy peer learner would be helpful. To address the challenge, we theoretically justify that distilling knowledge from a peer learner will lead to policy improvement and propose a disadvantageous distillation strategy based on the theoretical results. The conducted experiments on several continuous control tasks show that the proposed framework achieves superior performance with a learning-based agent and function approximation without the use of expensive teacher models.