mlsanityData to 7 Oct 2026

HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning

arXiv 2405.18080 ↗First posted 28 May 2024cs.LGcited by 14

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

The purpose of offline multi-task reinforcement learning (MTRL) is to develop a unified policy applicable to diverse tasks without the need for online environmental interaction. Recent advancements approach this through sequence modeling, leveraging the Transformer architecture's scalability and the benefits of parameter sharing to exploit task similarities. However, variations in task content and complexity pose significant challenges in policy formulation, necessitating judicious parameter sharing and management of conflicting gradients for optimal policy performance. In this work, we introduce the Harmony Multi-Task Decision Transformer (HarmoDT), a novel solution designed to identify an optimal harmony subspace of parameters for each task. We approach this as a bi-level optimization problem, employing a meta-learning framework that leverages gradient-based techniques. The upper level of this framework is dedicated to learning a task-specific mask that delineates the harmony subspace, while the inner level focuses on updating parameters to enhance the overall performance of the unified policy. Empirical evaluations on a series of benchmarks demonstrate the superiority of HarmoDT, verifying the effectiveness of our approach.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 60 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

43 of 60 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Q-value Regularized Transformer for Offline Reinforcement Learning27 May
  2. 2023Graph Decision Transformer7 Mar
  3. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  4. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  5. 2022Multi-Game Decision Transformers30 May
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun
  8. 2020Offline Meta-Reinforcement Learning with Advantage Weighting13 Aug

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
23 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  4. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  5. 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 441
  6. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  7. 2020Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropoutcited by 199
  8. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  9. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  10. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  11. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  12. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  13. 2023Recon: Reducing Conflicting Gradients from the Root for Multi-Task Learningcited by 50
  14. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  15. 2017Compression-aware Training of Deep Networkscited by 46
  16. 2024Sharing Knowledge in Multi-Task Deep Reinforcement Learningcited by 42
  17. 2022PaCo: Parameter-Compositional Multi-Task Reinforcement Learningcited by 40
  18. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 36
  19. 2020Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Controlcited by 27
  20. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  21. 2022FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregationcited by 16
  22. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  23. 2023Not All Tasks Are Equally Difficult: Multi-Task Deep Reinforcement Learning with Dynamic Depth Routingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 2; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Mastering Massive Multi-Task Reinforcement Learning via Mixture-of-Expert Decision Transformercited by 15
  2. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  3. 2024Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalizationcited by 6
  4. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  5. 2026DreOPD: Degraded-Reference Extrapolative On-Policy Distillation for Flow-matching Modelscited by 2
  6. 2024Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Plannercited by 1
  7. 2024Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 1
  8. 2024Continual Task Learning through Adaptive Policy Self-Compositioncited by 1
  9. 2026Mixture-of-World Models: Scaling Multi-Task Reinforcement Learning with Modular Latent Dynamicscited by 1
  10. 2024Communication Learning in Multi-Agent Systems from Graph Modeling Perspectivecited by 0
  11. 2025Target Return Optimizer for Multi-Game Decision Transformercited by 0
  12. 2026Conditional Sequence Modeling for Safe Reinforcement Learningcited by 0
2 more
  1. 2026WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimizationcited by 0
  2. 2026DaCe-DT: Data-Centric Offline Multi-Task Reinforcement Learning via Adaptive Prompts and Trajectory Correction for Heterogeneous Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learning
  2. 2024Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning
  3. 2026Behavior-Invariant Task Representation Learning with Transformer-based World Models for Offline Meta-Reinforcement Learning
  4. 2023HarmonyDream: Task Harmonization Inside World Models
  5. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  6. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning
  7. 2024Scrutinize What We Ignore: Reining In Task Representation Shift Of Context-Based Offline Meta Reinforcement Learning
  8. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought