Priors, Hierarchy, and Information Asymmetry for Skill Transfer in Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
The ability to discover behaviours from past experience and transfer them to new tasks is a hallmark of intelligent agents acting sample-efficiently in the real world. Equipping embodied reinforcement learners with the same ability may be crucial for their successful deployment in robotics. While hierarchical and KL-regularized reinforcement learning individually hold promise here, arguably a hybrid approach could combine their respective benefits. Key to these fields is the use of information asymmetry across architectural modules to bias which skills are learnt. While asymmetry choice has a large influence on transferability, existing methods base their choice primarily on intuition in a domain-independent, potentially sub-optimal, manner. In this paper, we theoretically and empirically show the crucial expressivity-transferability trade-off of skills across sequential tasks, controlled by information asymmetry. Given this insight, we introduce Attentive Priors for Expressive and Transferable Skills (APES), a hierarchical KL-regularized method, heavily benefiting from both priors and hierarchy. Unlike existing approaches, APES automates the choice of asymmetry by learning it in a data-driven, domain-dependent, way based on our expressivity-transferability theorems. Experiments over complex transfer domains of varying levels of extrapolation and sparsity, such as robot block stacking, demonstrate the criticality of the correct asymmetric choice, with APES drastically outperforming previous methods.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 62 references that are in the map. First posted on arXiv 20 Jan 2022.
Builds on
42 of 62 references.
References · 42
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
- 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2019Behavior Regularized Offline Reinforcement Learningcited by 654
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
- 2018Maximum a Posteriori Policy Optimisationcited by 318
- 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
30 more
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
- 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
- 2017Distral: Robust Multitask Reinforcement Learningcited by 231
- 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
- 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
- 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
- 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
- 2018Learning to Walk via Deep Reinforcement Learningcited by 153
- 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
- 2018Neural probabilistic motor primitives for humanoid controlcited by 96
- 2018Hierarchical Imitation and Reinforcement Learningcited by 88
- 2021From Motor Control to Team Play in Simulated Humanoid Footballcited by 87
- 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
- 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
- 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
- 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
- 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
- 2019Information asymmetry in KL-regularized RLcited by 53
- 2020Learning Off-Policy with Online Planningcited by 46
- 2020Behavior Priors for Efficient Reinforcement Learningcited by 40
- 2021Hierarchical Skills for Efficient Explorationcited by 37
- 2018Learning Abstract Optionscited by 31
- 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
- 2020Options of Interest: Temporal Abstraction with Interest Functionscited by 30
- 2021Learning Transferable Motor Skills with Hierarchical Latent Mixture Policiescited by 26
- 2020Latent Skill Planning for Exploration and Transfercited by 19
- 2019Exploiting Hierarchy for Learning and Transfer in KL-regularized RLcited by 15
- 2019Multitask Soft Option Learningcited by 13
- 2020Diversity-Enriched Option-Criticcited by 7
- 2019Attention-Privileged Reinforcement Learningcited by 4
Built on by
Citing papers per year: 2022: 2; 2023: 1.
3 papers.