mlsanityData to 13 Oct 2026

DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design

arXiv 2402.03479 ↗First posted 5 Feb 2024cs.LG cs.AIcited by 11

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Autonomous agents trained using deep reinforcement learning (RL) often lack the ability to successfully generalise to new environments, even when these environments share characteristics with the ones they have encountered during training. In this work, we investigate how the sampling of individual environment instances, or levels, affects the zero-shot generalisation (ZSG) ability of RL agents. We discover that, for deep actor-critic architectures sharing their base layers, prioritising levels according to their value loss minimises the mutual information between the agent's internal representation and the set of training levels in the generated training data. This provides a novel theoretical justification for the regularisation achieved by certain adaptive sampling strategies. We then turn our attention to unsupervised environment design (UED) methods, which assume control over level generation. We find that existing UED methods can significantly shift the training distribution, which translates to low ZSG performance. To prevent both overfitting and distributional shift, we introduce data-regularised environment design (DRED). DRED generates levels using a generative model trained to approximate the ground truth distribution of an initial set of level parameters. Through its grounding, DRED achieves significant improvements in ZSG over adaptive level sampling strategies and UED methods. Our code and experimental data are available at https://github.com/uoe-agents/dred.

Key facts. Cited by 11 papers in the map (data through 13 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 33 of its 47 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

33 of 47 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2018How Powerful are Graph Neural Networks?cited by 4,373
  3. 2013Auto-Encoding Variational Bayescited by 4,024
  4. 2015Continuous control with deep reinforcement learningcited by 3,290
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,411
  7. 2017Hindsight Experience Replaycited by 1,104
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
  9. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 704
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 576
  11. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 497
  12. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 479
21 more
  1. 2017Robust Adversarial Reinforcement Learningcited by 440
  2. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 399
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  4. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 302
  5. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 263
  6. 2020Prioritized Level Replaycited by 192
  7. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 171
  8. 2022Evolving Curricula with Regret-Based Environment Designcited by 168
  9. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  10. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  11. 2021Replay-Guided Adversarial Environment Designcited by 108
  12. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 104
  13. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  14. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  15. 2020OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasetscited by 73
  16. 2023MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learningcited by 35
  17. 2023Conditional Mutual Information for Disentangled Representations in Reinforcement Learningcited by 27
  18. 2019Learning to Optimize in Swarmscited by 23
  19. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 20
  20. 2020Instance based Generalization in Reinforcement Learningcited by 17
  21. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 6; 2025: 3; 2026: 2.

11 papers.

  1. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 42
  2. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  3. 2026Beyond Pixel Histories: World Models with Persistent 3D Statecited by 12
  4. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 9
  5. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  6. 2024Multi-view Disentanglement for Reinforcement Learning with Multiple Camerascited by 8
  7. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  8. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  9. 2025Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairscited by 2
  10. 2024Skill-aware Mutual Information Optimisation for Generalisation in Reinforcement Learningcited by 0
  11. 2026Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learningcited by 0