mlsanityData to 7 Oct 2026

Hierarchical Decision Transformer

arXiv 2209.10447 ↗First posted 21 Sep 2022cs.LG cs.AIcited by 16

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Sequence models in reinforcement learning require task knowledge to estimate the task policy. This paper presents a hierarchical algorithm for learning a sequence model from demonstrations. The high-level mechanism guides the low-level controller through the task by selecting sub-goals for the latter to reach. This sequence replaces the returns-to-go of previous methods, improving its performance overall, especially in tasks with longer episodes and scarcer rewards. We validate our method in multiple tasks of OpenAIGym, D4RL and RoboMimic benchmarks. Our method outperforms the baselines in eight out of ten tasks of varied horizons and reward frequencies without prior task knowledge, showing the advantages of the hierarchical model approach for learning from demonstrations using a sequence model.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 34 references that are in the map. First posted on arXiv 21 Sep 2022.

Builds on

20 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Online Decision Transformer11 Feb
  2. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016OpenAI Gymcited by 1,684
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  6. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  7. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  8. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  9. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  10. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  11. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  12. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
4 more
  1. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  2. 2019Learning from Trajectories via Subgoal Discoverycited by 27
  3. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 27
  4. 2017Learning Multi-Level Hierarchies with Hindsightcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 4; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  2. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  3. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  4. 2023A Survey of Demonstration Learningcited by 12
  5. 2023Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actionscited by 8
  6. 2025Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environmentscited by 6
  7. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
  8. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidancecited by 3
  9. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  10. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Modelcited by 1
  11. 2024Decision Mamba Architecturescited by 0
  12. 2025Adjusting the Output of Decision Transformer with Action Gradientcited by 0
4 more
  1. 2025Lessons from Neuroscience for AI: How integrating Actions, Compositional Structure and Episodic Memory could enable Safe, Interpretable and Human-Like AIcited by 0
  2. 2026In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Priorcited by 0
  3. 2026From Path Signatures to Sequential Modeling: Incremental Signature Contributions for Offline RLcited by 0
  4. 2026Intrinsic Vicarious Conditioning for Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2018Hierarchical Imitation and Reinforcement Learning
  2. 2021Towards Hierarchical Task Decomposition using Deep Reinforcement Learning for Pick and Place Subtasks
  3. 2023Hierarchical Imitation Learning with Vector Quantized Models
  4. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces
  5. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
  6. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  7. 2019Hierarchical Variational Imitation Learning of Control Programs
  8. 2022Deep Hierarchical Planning from Pixels