mlsanityData to 14 Oct 2026

Prompt Tuning Decision Transformers with Structured and Scalable Bandits

arXiv 2502.04979 ↗First posted 7 Feb 2025cs.LGcited by 2

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

also in 1 more
Abstract (arXiv)

Prompt tuning has emerged as a key technique for adapting large pre-trained Decision Transformers (DTs) in offline Reinforcement Learning (RL), particularly in multi-task and few-shot settings. The Prompting Decision Transformer (PDT) enables task generalization via trajectory prompts sampled uniformly from expert demonstrations -- without accounting for prompt informativeness. In this work, we propose a bandit-based prompt-tuning method that learns to construct optimal trajectory prompts from demonstration data at inference time. We devise a structured bandit architecture operating in the trajectory prompt space, achieving linear rather than combinatorial scaling with prompt size. Additionally, we show that the pre-trained PDT itself can serve as a powerful feature extractor for the bandit, enabling efficient reward modeling across various environments. We theoretically establish regret bounds and demonstrate empirically that our method consistently enhances performance across a wide range of tasks, high-dimensional environments, and out-of-distribution scenarios, outperforming existing baselines in prompt tuning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 37 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

22 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance1 Dec
  2. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer2 Aug
  3. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning31 Oct
  4. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  5. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptation17 Apr
  6. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun
  8. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 14

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  8. 2022A Generalist Agentcited by 808
  9. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  10. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  11. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  12. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
2 more
  1. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 17
  2. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Component-Based Survey of Interactions between Large Language Models and Multi-Armed Banditscited by 1
  2. 2026PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RL
  2. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning
  3. 2025In-Context Curiosity: Distilling Exploration for Decision-Pretrained Transformers on Bandit Tasks
  4. 2025Meta-Prompt Optimization for LLM-Based Sequential Decision Making
  5. 2025Prompting Decision Transformers for Zero-Shot Reach-Avoid Policies
  6. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
  7. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  8. 2024In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning