# Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillation

arXiv: https://arxiv.org/abs/2012.04839 · HTML: https://mlsanity.com/paper/2012.04839/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Stabilizing Unsupervised Environment Design with a Learned Adversary](https://mlsanity.com/paper/2308.10797/) (2023)
- [Guarded Policy Optimization with Imperfect Online Demonstrations](https://mlsanity.com/paper/2303.01728/) (2023)
- [Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomization](https://mlsanity.com/paper/2207.14561/) (2022)
- [Semantic-Aware Collaborative Deep Reinforcement Learning Over Wireless Cellular Networks](https://mlsanity.com/paper/2111.12064/) (2021)
- [The Curse of Diversity in Ensemble-Based Exploration](https://mlsanity.com/paper/2405.04342/) (2024)
- [Distilled Domain Randomization](https://mlsanity.com/paper/2112.03149/) (2021)
- [VIBR: Learning View-Invariant Value Functions for Robust Visual Control](https://mlsanity.com/paper/2306.08537/) (2023)
- [Representation Convergence: Mutual Distillation is Secretly a Form of Regularization](https://mlsanity.com/paper/2501.02481/) (2025)
- [Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Survey](https://mlsanity.com/paper/2602.00399/) (2026)

## Abstract (arXiv)

In reinforcement learning, domain randomisation is an increasingly popular technique for learning more general policies that are robust to domain-shifts at deployment. However, naively aggregating information from randomised domains may lead to high variance in gradient estimation and unstable learning process. To address this issue, we present a peer-to-peer online distillation strategy for RL termed P2PDRL, where multiple workers are each assigned to a different environment, and exchange knowledge through mutual regularisation based on Kullback-Leibler divergence. Our experiments on continuous control tasks show that P2PDRL enables robust learning across a wider randomisation distribution than baselines, and more robust generalisation to new environments at testing.