# Hierarchical Reinforcement Learning via Advantage-Weighted Information Maximization

arXiv: https://arxiv.org/abs/1901.01365 · HTML: https://mlsanity.com/paper/1901.01365/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jan 2019.

## Built on by (most cited first, 6 of 6)

- [Reparameterized Policy Learning for Multimodal Trajectory Optimization](https://mlsanity.com/paper/2307.10710/) (2023)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [Motion Planning by Learning the Solution Manifold in Trajectory Optimization](https://mlsanity.com/paper/2107.05842/) (2021)
- [Hierarchical Reinforcement Learning with Optimal Level Synchronization Based on Flow-Based Deep Generative Model](https://mlsanity.com/paper/2107.08183/) (2021)
- [Learning Temporal Abstractions via Variational Homomorphisms in Option-Induced Abstract MDPs](https://mlsanity.com/paper/2507.16473/) (2025)
- [STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order](https://mlsanity.com/paper/2601.08107/) (2026)

## Abstract (arXiv)

Real-world tasks are often highly structured. Hierarchical reinforcement learning (HRL) has attracted research interest as an approach for leveraging the hierarchical structure of a given task in reinforcement learning (RL). However, identifying the hierarchical policy structure that enhances the performance of RL is not a trivial task. In this paper, we propose an HRL method that learns a latent variable of a hierarchical policy using mutual information maximization. Our approach can be interpreted as a way to learn a discrete and latent representation of the state-action space. To learn option policies that correspond to modes of the advantage function, we introduce advantage-weighted importance sampling. In our HRL method, the gating policy learns to select option policies based on an option-value function, and these option policies are optimized based on the deterministic policy gradient method. This framework is derived by leveraging the analogy between a monolithic policy in standard RL and a hierarchical policy in HRL by using a deterministic option policy. Experimental results indicate that our HRL approach can learn a diversity of options and that it can enhance the performance of RL in continuous control tasks.