POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
The goal of Unsupervised Reinforcement Learning (URL) is to find a reward-agnostic prior policy on a task domain, such that the sample-efficiency on supervised downstream tasks is improved. Although agents initialized with such a prior policy can achieve a significantly higher reward with fewer samples when finetuned on the downstream task, it is still an open question how an optimal pretrained prior policy can be achieved in practice. In this work, we present POLTER (Policy Trajectory Ensemble Regularization) - a general method to regularize the pretraining that can be applied to any URL algorithm and is especially useful on data- and knowledge-based URL algorithms. It utilizes an ensemble of policies that are discovered during pretraining and moves the policy of the URL algorithm closer to its optimal prior. Our method is based on a theoretical framework, and we analyze its practical effects on a white-box benchmark, allowing us to study POLTER with full control. In our main experiments, we evaluate POLTER on the Unsupervised Reinforcement Learning Benchmark (URLB), which consists of 12 tasks in 3 domains. We demonstrate the generality of our approach by improving the performance of a diverse set of data- and knowledge-based URL algorithms by 19% on average and up to 40% in the best case. Under a fair comparison with tuned baselines and tuned POLTER, we establish a new state-of-the-art for model-free methods on the URLB.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 66 references that are in the map. First posted on arXiv 23 May 2022.
Builds on
41 of 66 references.
References · 41
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018World Modelscited by 1,513
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
- 2018Exploration by Random Network Distillationcited by 993
- 2018DeepMind Control Suitecited by 956
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
- 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
- 2019Solving Rubik's Cube with a Robot Handcited by 734
29 more
- 2022Temporal Difference Learning for Model Predictive Controlcited by 397
- 2020dm_control: Software and Tasks for Continuous Controlcited by 375
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
- 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
- 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
- 2016Variational Intrinsic Controlcited by 267
- 2019Self-Supervised Exploration via Disagreementcited by 265
- 2018Provably Efficient Maximum Entropy Explorationcited by 242
- 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
- 2019Efficient Exploration via State Marginal Matchingcited by 188
- 2021Reinforcement Learning with Prototypical Representationscited by 179
- 2020Reward-Free Exploration for Reinforcement Learningcited by 177
- 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
- 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
- 2021APS: Active Pretraining with Successor Featurescited by 112
- 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 108
- 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
- 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
- 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
- 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 68
- 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
- 2020Relative Variational Intrinsic Controlcited by 34
- 2021Geometric Entropic Explorationcited by 32
- 2021Wasserstein Unsupervised Reinforcement Learningcited by 22
- 2021Unsupervised Reinforcement Learning in Multiple Environmentscited by 17
- 2022One After Another: Learning Incremental Skills for a Changing Worldcited by 10
- 2014An Adversarial Interpretation of Information-Theoretic Bounded Rationalitycited by 6
- 2022Unsupervised Reinforcement Learning for Transferable Manipulation Skill Discoverycited by 5
- 2022Nuclear Norm Maximization Based Curiosity-Driven Learningcited by 4
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.