# Hyper-Decision Transformer for Efficient Online Policy Adaptation

arXiv: https://arxiv.org/abs/2304.08487 · HTML: https://mlsanity.com/paper/2304.08487/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 35 of its 39 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks](https://mlsanity.com/paper/2106.04489/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Prompting Decision Transformer for Few-Shot Policy Generalization](https://mlsanity.com/paper/2206.13499/) (2022)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)

## Built on by (most cited first, 30 of 33)

- [Supervised Pretraining Can Learn In-Context Reinforcement Learning](https://mlsanity.com/paper/2306.14892/) (2023)
- [In-Context Imitation Learning via Next-Token Prediction](https://mlsanity.com/paper/2408.15980/) (2024)
- [Elastic Decision Transformer](https://mlsanity.com/paper/2307.02484/) (2023)
- [TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models](https://mlsanity.com/paper/2310.05905/) (2023)
- [Generalization to New Sequential Decision Making Tasks with In-Context Learning](https://mlsanity.com/paper/2312.03801/) (2023)
- [Learning to Modulate pre-trained Models in RL](https://mlsanity.com/paper/2306.14884/) (2023)
- [Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging](https://mlsanity.com/paper/2512.08333/) (2025)
- [One-Shot Imitation Learning with Invariance Matching for Robotic Manipulation](https://mlsanity.com/paper/2405.13178/) (2024)
- [Reinforcement Learning with Token-level Feedback for Controllable Text Generation](https://mlsanity.com/paper/2403.11558/) (2024)
- [Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion](https://mlsanity.com/paper/2407.10973/) (2024)
- [Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Loss](https://mlsanity.com/paper/2402.06187/) (2024)
- [Dynamics as Prompts: In-Context Learning for Sim-to-Real System Identifications](https://mlsanity.com/paper/2410.20357/) (2024)
- [Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers](https://mlsanity.com/paper/2410.24108/) (2024)
- [Dynamic Mixture of Progressive Parameter-Efficient Expert Library for Lifelong Robot Learning](https://mlsanity.com/paper/2506.05985/) (2025)
- [Solving Continual Offline Reinforcement Learning with Decision Transformer](https://mlsanity.com/paper/2401.08478/) (2024)
- [Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer](https://mlsanity.com/paper/2408.01402/) (2024)
- [The Duality of Generative AI and Reinforcement Learning in Robotics: A Review](https://mlsanity.com/paper/2410.16411/) (2024)
- [Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization](https://mlsanity.com/paper/2411.01168/) (2024)
- [Decision Transformer as a Foundation Model for Partially Observable Continuous Control](https://mlsanity.com/paper/2404.02407/) (2024)
- [NeurDB: An AI-powered Autonomous Data System](https://mlsanity.com/paper/2405.03924/) (2024)
- [HyperPPO: A scalable method for finding small policies for robotic control](https://mlsanity.com/paper/2309.16663/) (2023)
- [Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction](https://mlsanity.com/paper/2402.04154/) (2024)
- [Adversarially Robust Decision Transformer](https://mlsanity.com/paper/2407.18414/) (2024)
- [Meta-Controller: Few-Shot Imitation of Unseen Embodiments and Tasks in Continuous Control](https://mlsanity.com/paper/2412.12147/) (2024)
- [Physics-informed Imitative Reinforcement Learning for Real-world Driving](https://mlsanity.com/paper/2407.02508/) (2024)
- [Prompt Tuning Decision Transformers with Structured and Scalable Bandits](https://mlsanity.com/paper/2502.04979/) (2025)
- [Accelerating Neural Self-Improvement via Bootstrapping](https://mlsanity.com/paper/2305.01547/) (2023)
- [Task-agnostic Decision Transformer for Multi-type Agent Control with Federated Split Training](https://mlsanity.com/paper/2405.13445/) (2024)
- [DRDT3: Diffusion-Refined Decision Test-Time Training Model](https://mlsanity.com/paper/2501.06718/) (2025)
- [Target Return Optimizer for Multi-Game Decision Transformer](https://mlsanity.com/paper/2503.02311/) (2025)

## Abstract (arXiv)

Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT to efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT's generalization capability on object manipulation tasks. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore a more challenging setting where expert actions are not available, and we show that HDT outperforms state-of-the-art baselines in terms of task success rates by a large margin.