mlsanityData to 7 Oct 2026

Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance

arXiv 2412.00979 ↗First posted 1 Dec 2024cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision transformers recast reinforcement learning as a conditional sequence generation problem, offering a simple but effective alternative to traditional value or policy-based methods. A recent key development in this area is the integration of prompting in decision transformers to facilitate few-shot policy generalization. However, current methods mainly use static prompt segments to guide rollouts, limiting their ability to provide context-specific guidance. Addressing this, we introduce a hierarchical prompting approach enabled by retrieval augmentation. Our method learns two layers of soft tokens as guiding prompts: (1) global tokens encapsulating task-level information about trajectories, and (2) adaptive tokens that deliver focused, timestep-specific instructions. The adaptive tokens are dynamically retrieved from a curated set of demonstration segments, ensuring context-aware guidance. Experiments across seven benchmark tasks in the MuJoCo and MetaWorld environments demonstrate the proposed approach consistently outperforms all baseline methods, suggesting that hierarchical prompting for decision transformers is an effective strategy to enable few-shot policy generalization.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 41 of its 55 references that are in the map. First posted on arXiv 1 Dec 2024.

Builds on

41 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining12 Oct
  2. 2022Hierarchical Decision Transformer21 Sep
  3. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  4. 2022Bootstrapped Transformer for Offline Reinforcement Learning17 Jun
  5. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments31 May
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun
  9. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 32

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2017Deep Setscited by 1,394
  7. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  8. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  10. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
20 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  2. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  3. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  4. 2017Meta Networkscited by 268
  5. 2019Time2Vec: Learning a Vector Representation of Timecited by 208
  6. 2018Delete, Retrieve, Generate: A Simple Approach to Sentiment and Style Transfercited by 191
  7. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  8. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  9. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  10. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  11. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  12. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  13. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  14. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 65
  15. 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
  16. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 34
  17. 2019Text Generation with Exemplar-based Adaptive Decodingcited by 28
  18. 2022Unsupervised Vision-and-Language Pre-training via Retrieval-based Multi-Granular Alignmentcited by 23
  19. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  20. 2021ST-MAML: A Stochastic-Task based Method for Task-Heterogeneous Meta-Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Prompt Tuning Decision Transformers with Structured and Scalable Banditscited by 2
  2. 2026Large Language Model-Powered Evolutionary Code Optimization on a Phylogenetic Treecited by 2
  3. 2026Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Auditcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  2. 2025Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RL
  3. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
  4. 2024A Minimalist Prompt for Zero-Shot Policy Learning
  5. 2025LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning
  6. 2025Dynamic Policy Induction for Adaptive Prompt Optimization: Bridging the Efficiency-Accuracy Gap via Lightweight Reinforcement Learning
  7. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization
  8. 2025Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs