mlsanityData to 7 Oct 2026

State-Conditioned Adversarial Subgoal Generation

arXiv 2201.09635 ↗First posted 24 Jan 2022cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Hierarchical reinforcement learning (HRL) proposes to solve difficult tasks by performing decision-making and control at successively higher levels of temporal abstraction. However, off-policy HRL often suffers from the problem of a non-stationary high-level policy since the low-level policy is constantly changing. In this paper, we propose a novel HRL approach for mitigating the non-stationarity by adversarially enforcing the high-level policy to generate subgoals compatible with the current instantiation of the low-level policy. In practice, the adversarial learning is implemented by training a simple state-conditioned discriminator network concurrently with the high-level policy which determines the compatibility level of subgoals. Comparison to state-of-the-art algorithms shows that our approach improves both learning efficiency and performance in challenging continuous control tasks.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 16 of its 27 references that are in the map. First posted on arXiv 24 Jan 2022.

Builds on

16 of 27 references.

References · 16

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2014Conditional Generative Adversarial Netscited by 2,152
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  5. 2016The Option-Critic Architecturecited by 433
  6. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  7. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  8. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  9. 2016Variational Intrinsic Controlcited by 262
  10. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  11. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  12. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
4 more
  1. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  2. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 55
  3. 2019Exploration via Hindsight Goal Generationcited by 53
  4. 2017Multi-Level Discovery of Deep Optionscited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 1; 2026: 1.

8 papers.

  1. 2024Probabilistic Subgoal Representations for Hierarchical Reinforcement learningcited by 8
  2. 2025Hierarchical Reinforcement Learning with Uncertainty-Guided Diffusional Subgoalscited by 5
  3. 2023CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learningcited by 2
  4. 2023Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rolloutcited by 2
  5. 2024Training Fast Robot Policies with Slow Foundation Modelscited by 2
  6. 2024Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approachcited by 1
  7. 2023Landmark Guided Active Exploration with State-specific Balance Coefficientcited by 0
  8. 2026Learning Multi-Timescale Abstractions for Hierarchical Combinatorial Planningcited by 0