mlsanityData to 7 Oct 2026

Rethinking Decision Transformer via Hierarchical Reinforcement Learning

arXiv 2311.00267 ↗First posted 1 Nov 2023cs.LG cs.AIcited by 14

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformer (DT) is an innovative algorithm leveraging recent advances of the transformer architecture in reinforcement learning (RL). However, a notable limitation of DT is its reliance on recalling trajectories from datasets, losing the capability to seamlessly stitch sub-optimal trajectories together. In this work we introduce a general sequence modeling framework for studying sequential decision making through the lens of Hierarchical RL. At the time of making decisions, a high-level policy first proposes an ideal prompt for the current state, a low-level policy subsequently generates an action conditioned on the given prompt. We show DT emerges as a special case of this framework with certain choices of high-level and low-level policies, and discuss the potential failure of these choices. Inspired by these observations, we study how to jointly optimize the high-level and low-level policies to enable the stitching ability, which further leads to the development of new offline RL algorithms. Our empirical results clearly show that the proposed algorithms significantly surpass DT on several control and navigation benchmarks. We hope our contributions can inspire the integration of transformer architectures within the field of RL.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 54 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

45 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions18 Sep
  2. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets24 Jun
  3. 2022Hierarchical Decision Transformer21 Sep
  4. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  5. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
27 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2022A Generalist Agentcited by 802
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  10. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  13. 2016The Option-Critic Architecturecited by 433
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  15. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  16. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  17. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  18. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  19. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  20. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  21. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  22. 2023Elastic Decision Transformercited by 66
  23. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  24. 2022Settling the Reward Hypothesiscited by 33
  25. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  26. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  27. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 4; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  2. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
  3. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  4. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  5. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
  6. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learningcited by 2
  7. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  8. 2024SANDWICH: Towards an Offline, Differentiable, Fully-Trainable Wireless Neural Ray-Tracing Surrogatecited by 1
  9. 2024Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  10. 2025Adjusting the Output of Decision Transformer with Action Gradientcited by 0
  11. 2025Robust Adversarial Reinforcement Learning in Stochastic Games via Sequence Modelingcited by 0
  12. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
2 more
  1. 2026Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  2. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance
  2. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
  3. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
  4. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  5. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation
  6. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem
  8. 2024Decision Mamba: Reinforcement Learning via Sequence Modeling with Selective State Spaces