mlsanityData to 14 Oct 2026

How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning

arXiv 2505.16581 ↗First posted 22 May 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In the zero-shot policy transfer setting in reinforcement learning, the goal is to train an agent on a fixed set of training environments so that it can generalise to similar, but unseen, testing environments. Previous work has shown that policy distillation after training can sometimes produce a policy that outperforms the original in the testing environments. However, it is not yet entirely clear why that is, or what data should be used to distil the policy. In this paper, we prove, under certain assumptions, a generalisation bound for policy distillation after training. The theory provides two practical insights: for improved generalisation, you should 1) train an ensemble of distilled policies, and 2) distil it on as much data from the training environments as possible. We empirically verify that these insights hold in more general settings, when the assumptions required for the theory no longer hold. Finally, we demonstrate that an ensemble of policies distilled on a diverse dataset can generalise significantly better than the original agent.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 44 of its 60 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

44 of 60 references.

References · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  8. 2015Policy Distillationcited by 440
  9. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  12. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
32 more
  1. 2017Distral: Robust Multitask Reinforcement Learningcited by 233
  2. 2015Contextual Markov Decision Processescited by 220
  3. 2018Why do deep convolutional networks generalize so poorly to small image transformations?cited by 198
  4. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
  5. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  6. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  7. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  8. 2019Distilling Policy Distillationcited by 84
  9. 2018Kickstarting Deep Reinforcement Learningcited by 81
  10. 2020On the Benefits of Invariance in Neural Networkscited by 79
  11. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  12. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  13. 2017Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physicscited by 58
  14. 2017Divide-and-Conquer Reinforcement Learningcited by 48
  15. 2020On the Generalization Effects of Linear Transformations in Data Augmentationcited by 47
  16. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 43
  17. 2022How Much Data Are Augmentations Worth? An Investigation into Scaling Laws, Invariance, and Implicit Regularizationcited by 40
  18. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  19. 2021Unsupervised Visual Attention and Invariance for Reinforcement Learningcited by 30
  20. 2023The Generalization Gap in Offline Reinforcement Learningcited by 26
  21. 2022The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspectivecited by 24
  22. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  23. 2023Optimization Dynamics of Equivariant and Augmented Neural Networkscited by 18
  24. 2024Emergent Equivariance in Deep Ensemblescited by 18
  25. 2022Learning Instance-Specific Augmentations by Capturing Local Invariancescited by 15
  26. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  27. 2022In What Ways Are Deep Neural Networks Invariant and How Should We Measure This?cited by 9
  28. 2024Ensembles provably learn equivariance through data augmentationcited by 8
  29. 2018Multi-task Learning for Continuous Controlcited by 6
  30. 2022Tight Performance Guarantees of Imitator Policies with Continuous Actionscited by 6
  31. 2017Collaborative Deep Reinforcement Learningcited by 5
  32. 2024Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Distilling Reinforcement Learning into Single-Batch Datasets
  2. 2026Understanding Goal Generalisation in Sequential Reinforcement Learning
  3. 2024Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning
  4. 2024Proximal Policy Distillation
  5. 2026Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance
  6. 2024Behaviour Distillation
  7. 2025Representation Convergence: Mutual Distillation is Secretly a Form of Regularization
  8. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing