mlsanityData to 7 Oct 2026

Prompt-Tuning Bandits: Enabling Few-Shot Generalization for Efficient Multi-Task Offline RL

arXiv 2502.06358 ↗First posted 10 Feb 2025cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Prompting has emerged as the dominant paradigm for adapting large, pre-trained transformer-based models to downstream tasks. The Prompting Decision Transformer (PDT) enables large-scale, multi-task offline Reinforcement Learning (RL) pre-training by leveraging stochastic trajectory prompts to identify the target task. However, these prompts are sampled uniformly from expert demonstrations, overlooking a critical limitation: not all prompts are equally informative for differentiating between tasks. This limits generalization and adaptation, especially in low-data or open-world settings where sample efficiency is crucial. To address this issue, we propose a lightweight, inference-time, bandit-based prompt-tuning framework. The bandit explores and optimizes trajectory prompt selection to enhance task performance, while avoiding costly fine-tuning of the transformer backbone. Our experiments indicate not only clear performance gains due to bandit-based prompt-tuning, but also better sample complexity, scalability, and prompt space exploration compared to prompt-tuning baselines. These results highlights the importance of adaptive prompt selection mechanisms for efficient generalization in offline multi-task RL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 13 of its 21 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

13 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 11

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2022A Generalist Agentcited by 802
  7. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68
  8. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  9. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 34
  10. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  11. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompt Tuning Decision Transformers with Structured and Scalable Bandits
  2. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
  3. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidance
  4. 2026Improving Zero-Shot Offline RL via Behavioral Task Sampling
  5. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models
  6. 2025LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learning
  7. 2024A Minimalist Prompt for Zero-Shot Policy Learning
  8. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?