mlsanityData to 7 Oct 2026

Hyper-Decision Transformer for Efficient Online Policy Adaptation

arXiv 2304.08487 ↗First posted 17 Apr 2023cs.LG cs.AI cs.CVcited by 32

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT to efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT's generalization capability on object manipulation tasks. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore a more challenging setting where expert actions are not available, and we show that HDT outperforms state-of-the-art baselines in terms of task success rates by a large margin.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 39 references that are in the map. First posted on arXiv 17 Apr 2023.

Builds on

35 of 39 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  2. 2022Multi-Game Decision Transformers30 May
  3. 2022Online Decision Transformer11 Feb
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
17 more
  1. 2016HyperNetworkscited by 1,003
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2022A Generalist Agentcited by 802
  5. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  6. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  7. 2019Continual learning with hypernetworkscited by 267
  8. 2017One-Shot Imitation Learningcited by 232
  9. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 225
  10. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  11. 2019Recent Advances in Imitation Learning from Observationcited by 92
  12. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  13. 2020Transformers for One-Shot Visual Imitationcited by 77
  14. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 65
  15. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  16. 2022Skill-based Meta-Reinforcement Learningcited by 36
  17. 2021HyperDynamics: Meta-Learning Object and Agent Dynamics with Hypernetworkscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 18; 2025: 6; 2026: 1.

32 papers.

Sort:

By area:

  1. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  2. 2023Elastic Decision Transformercited by 66
  3. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  4. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  5. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  6. 2023Learning to Modulate pre-trained Models in RLcited by 28
  7. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 19
  8. 2024One-Shot Imitation Learning with Invariance Matching for Robotic Manipulationcited by 18
  9. 2024Reinforcement Learning with Token-level Feedback for Controllable Text Generationcited by 16
  10. 2024Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusioncited by 12
  11. 2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
  12. 2024Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identificationscited by 10
20 more
  1. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  2. 2025Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learningcited by 8
  3. 2024Solving Continual Offline Reinforcement Learning with Decision Transformercited by 6
  4. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  5. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6
  6. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  7. 2024NeurDB: An AI-powered Autonomous Data Systemcited by 5
  8. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  9. 2023HyperPPO: A scalable method for finding small policies for robotic controlcited by 4
  10. 2024Read to Play (R2-Play): Decision Transformer with Multimodal Game Instructioncited by 4
  11. 2024Adversarially Robust Decision Transformercited by 4
  12. 2024Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Controlcited by 3
  13. 2024Physics-informed Imitative Reinforcement Learning for Real-world Drivingcited by 2
  14. 2025Prompt Tuning Decision Transformers with Structured and Scalable Banditscited by 2
  15. 2023Accelerating Neural Self-Improvement via Bootstrappingcited by 1
  16. 2024Task-agnostic Decision Transformer for Multi-type Agent Control with Federated Split Trainingcited by 1
  17. 2025DRDT3: Diffusion-Refined Decision Test-Time Training Modelcited by 1
  18. 2025Target Return Optimizer for Multi-Game Decision Transformercited by 0
  19. 2025Robust Adversarial Reinforcement Learning in Stochastic Games via Sequence Modelingcited by 0
  20. 2026TabQL: In-Context Q-Learning with Tabular Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  2. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  3. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  4. 2024Return-Aligned Decision Transformer
  5. 2026Decoupling Return-to-Go for Efficient Decision Transformer
  6. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
  7. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance
  8. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining