mlsanityData to 7 Oct 2026

Hierarchical Imitation Learning with Vector Quantized Models

arXiv 2301.12962 ↗First posted 30 Jan 2023cs.AI cs.LGcited by 15

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

The ability to plan actions on multiple levels of abstraction enables intelligent agents to solve complex tasks effectively. However, learning the models for both low and high-level planning from demonstrations has proven challenging, especially with higher-dimensional inputs. To address this issue, we propose to use reinforcement learning to identify subgoals in expert trajectories by associating the magnitude of the rewards with the predictability of low-level actions given the state and the chosen subgoal. We build a vector-quantized generative model for the identified subgoals to perform subgoal-level planning. In experiments, the algorithm excels at solving complex, long-horizon decision-making problems outperforming state-of-the-art. Because of its ability to plan, our algorithm can find better trajectories than the ones in the training set

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 73 references that are in the map. First posted on arXiv 30 Jan 2023.

Builds on

46 of 73 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning16 Dec
  2. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  3. 2021Adversarial Option-Aware Hierarchical Imitation Learning10 Jun
  4. 2020Provable Hierarchical Imitation Learning via EM7 Oct
  5. 2018An Algorithmic Perspective on Imitation Learning16 Nov
  6. 2018Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information29 Sep

Other references · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2017Neural Discrete Representation Learningcited by 4,650
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2016The Option-Critic Architecturecited by 433
  11. 2021Mastering Atari Games with Limited Datacited by 235
  12. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
28 more
  1. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  2. 2022Learning from All Vehiclescited by 135
  3. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  4. 2020Model-Based Offline Planningcited by 126
  5. 2021Implicit MLE: Backpropagating Through Discrete Exponential Family Distributionscited by 97
  6. 2022Deep Hierarchical Planning from Pixelscited by 96
  7. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  8. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  9. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  10. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  11. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  12. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  13. 2018Relational Deep Reinforcement Learningcited by 72
  14. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 66
  15. 2019An investigation of model-free planningcited by 64
  16. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  17. 2020Robust Training of Vector Quantized Bottleneck Modelscited by 53
  18. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  19. 2021Vector Quantized Models for Planningcited by 47
  20. 2017OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learningcited by 40
  21. 2018Time-Agnostic Prediction: Predicting Predictable Video Framescited by 37
  22. 2019Learning from Trajectories via Subgoal Discoverycited by 27
  23. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 27
  24. 2018Single-Agent Policy Tree Search With Guaranteescited by 22
  25. 2021Subgoal Search For Complex Reasoning Taskscited by 21
  26. 2021Policy-Guided Heuristic Search with Guaranteescited by 20
  27. 2022Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Searchcited by 12
  28. 2021Explainable Hierarchical Imitation Learning for Robotic Drink Pouringcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 2; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Controlcited by 21
  2. 2023Chain-of-Thought Predictive Controlcited by 18
  3. 2023Efficient Planning with Latent Diffusioncited by 11
  4. 2023Hieros: Hierarchical Imagination on Structured State Space Sequence World Modelscited by 8
  5. 2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
  6. 2024Predicting Long-Term Human Behaviors in Discrete Representations via Physics-Guided Diffusioncited by 6
  7. 2024BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulationcited by 5
  8. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  9. 2025OptionZero: Planning with Learned Optionscited by 2
  10. 2024What Matters in Hierarchical Search for Combinatorial Reasoning Problems?cited by 1
  11. 2025Subgoal-Guided Policy Heuristic Search with Learned Subgoalscited by 1
  12. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Ordercited by 0
3 more
  1. 2026Primary-Fine Decoupling for Action Generation in Robotic Imitationcited by 0
  2. 2026Structure-Induced Information for Rerooting Levin Tree Searchcited by 0
  3. 2026Implicit Neural Representations of Individual Behaviorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Structured Imitation Learning of Interactive Policies through Inverse Games
  2. 2019Hierarchical Variational Imitation Learning of Control Programs
  3. 2019Reward-Conditioned Policies
  4. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations
  5. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
  6. 2022Hierarchical Decision Transformer
  7. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning
  8. 2020State-only Imitation with Transition Dynamics Mismatch