mlsanityData to 14 Oct 2026

SHIRO: Soft Hierarchical Reinforcement Learning

arXiv 2212.12786 ↗First posted 24 Dec 2022cs.RO cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Hierarchical Reinforcement Learning (HRL) algorithms have been demonstrated to perform well on high-dimensional decision making and robotic control tasks. However, because they solely optimize for rewards, the agent tends to search the same space redundantly. This problem reduces the speed of learning and achieved reward. In this work, we present an Off-Policy HRL algorithm that maximizes entropy for efficient exploration. The algorithm learns a temporally abstracted low-level policy and is able to explore broadly through the addition of entropy to the high-level. The novelty of this work is the theoretical motivation of adding entropy to the RL objective in the HRL setting. We empirically show that the entropy can be added to both levels if the Kullback-Leibler (KL) divergence between consecutive updates of the low-level policy is sufficiently small. We performed an ablative study to analyze the effects of entropy on hierarchy, in which adding entropy to high-level emerged as the most desirable configuration. Furthermore, a higher temperature in the low-level leads to Q-value overestimation and increases the stochasticity of the environment that the high-level operates on, making learning more challenging. Our method, SHIRO, surpasses state-of-the-art performance on a range of simulated robotic control benchmark tasks and requires minimal tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 24 Dec 2022.

Builds on

40 of 64 references.

References · 40

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  7. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  9. 2016The Option-Critic Architecturecited by 448
  10. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
28 more
  1. 2018Maximum a Posteriori Policy Optimisationcited by 328
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  3. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  4. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  5. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  6. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
  7. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 108
  8. 2017Meta Learning Shared Hierarchiescited by 93
  9. 2016A Deep Hierarchical Approach to Lifelong Learning in Minecraftcited by 90
  10. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  11. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  12. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  13. 2019Data Efficient Reinforcement Learning for Legged Robotscited by 63
  14. 2016Learning and Transfer of Modulated Locomotor Controllerscited by 59
  15. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  16. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  17. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  18. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  19. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  20. 2019Multi-Agent Manipulation via Locomotion using Hierarchical Sim2Realcited by 42
  21. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 40
  22. 2019Hierarchical Reinforcement Learning with Advantage-Based Auxiliary Rewardscited by 40
  23. 2019Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulationcited by 35
  24. 2020Hierarchically Decoupled Imitation for Morphological Transfercited by 31
  25. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 30
  26. 2019Hierarchical Reinforcement Learning for Quadruped Locomotioncited by 24
  27. 2018Policy Search in Continuous Action Domains: an Overviewcited by 10
  28. 2019Hierarchical Soft Actor-Critic: Adversarial Exploration via Mutual Information Optimizationcited by 2

Built on by

Not cited yet.