mlsanityData to 14 Oct 2026

CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learning

arXiv 2304.03535 ↗First posted 7 Apr 2023cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) leverages temporal abstraction to efficiently tackle complex long-horizon tasks. However, HRL often collapses because the continual updates of the low-level primitive make earlier sub-goals issued by the high-level policy obsolete, introducing non-stationarity that destabilizes training. We propose CRISP, a curriculum-driven framework that tackles this instability with three key ingredients: (1) primitive-informed parsing (PIP), which adaptively re-labels a handful of expert demonstrations to always generate reachable subgoals by the current low-level primitive, (2) an inverse-reinforcement-learning regularizer that steers the high-level policy toward the expert-induced subgoal distribution and stabilizes learning, and (3) a unified training loop that leverages these components to boost sample efficiency. Across six sparse-reward robotic navigation and manipulation benchmarks, CRISP improves success rates by more than 40% over strong hierarchical and flat baselines and successfully transfers to real-world tasks, demonstrating the promise of curriculum-based HRL for practical scenarios.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 48 of its 70 references that are in the map. First posted on arXiv 7 Apr 2023.

Builds on

48 of 70 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  8. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  9. 2016The Option-Critic Architecturecited by 448
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  12. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
36 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  3. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  4. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  5. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  6. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  8. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  10. 2019Goal-conditioned Imitation Learningcited by 162
  11. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  13. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 90
  14. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  15. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  16. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  17. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 67
  18. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 65
  19. 2018TACO: Learning Task Decomposition via Temporal Alignment for Controlcited by 62
  20. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  21. 2019Exploration via Hindsight Goal Generationcited by 54
  22. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  23. 2017Multi-Level Discovery of Deep Optionscited by 51
  24. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  25. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 33
  26. 2020Data-efficient Hindsight Off-policy Option Learningcited by 31
  27. 2017Deep Q-learning from Demonstrationscited by 28
  28. 2019Automated curricula through setter-solver interactionscited by 26
  29. 2017Learnings Options End-to-End for Continuous Action Taskscited by 24
  30. 2019The Termination Criticcited by 24
  31. 2017Learning Multi-Level Hierarchies with Hindsightcited by 20
  32. 2018Accuracy-based Curriculum Learning in Deep Reinforcement Learningcited by 15
  33. 2022Robust Reinforcement Learning via Genetic Curriculumcited by 10
  34. 2022State-Conditioned Adversarial Subgoal Generationcited by 8
  35. 2017Learning with Options that Terminate Off-Policycited by 7
  36. 2019Compositional Transfer in Hierarchical Reinforcement Learningcited by 7

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approachcited by 1
  2. 2025Sample-Efficient Robot Skill Learning for Construction Tasks: Benchmarking Hierarchical Reinforcement Learning and Vision-Language-Action VLA Modelcited by 0