mlsanityData to 7 Oct 2026

Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policies

arXiv 2303.07551 ↗First posted 14 Mar 2023cs.LG cs.AIcited by 15

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recent work has shown the promise of creating generalist, transformer-based, models for language, vision, and sequential decision-making problems. To create such models, we generally require centralized training objectives, data, and compute. It is of interest if we can more flexibly create generalist policies by merging together multiple, task-specific, individually trained policies. In this work, we take a preliminary step in this direction through merging, or averaging, subsets of Decision Transformers in parameter space trained on different MuJoCo locomotion problems, forming multi-task models without centralized training. We also demonstrate the importance of various methodological choices when merging policies, such as utilizing common pre-trained initializations, increasing model capacity, and utilizing Fisher information for weighting parameter importance. In general, we believe research in this direction could help democratize and distribute the process that forms multi-task robotics policies. Our implementation is available at https://github.com/daniellawson9999/merging-decision-transformers.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 42 references that are in the map. First posted on arXiv 14 Mar 2023.

Builds on

35 of 42 references.

Closest ancestors · 5

Same direction, newest first

  1. 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning17 Nov
  2. 2022Multi-Game Decision Transformers30 May
  3. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2016Layer Normalizationcited by 5,579
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  8. 2022Editing Models with Task Arithmeticcited by 1,076
  9. 2022A Generalist Agentcited by 802
  10. 2021Robust fine-tuning of zero-shot modelscited by 785
  11. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  12. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
18 more
  1. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  2. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  3. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  4. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 268
  5. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  6. 2022Patching open-vocabulary models by interpolating weightscited by 187
  7. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  8. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  9. 2022Fusing finetuned models for better pretrainingcited by 110
  10. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  11. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  12. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  13. 2022Formal Algorithms for Transformerscited by 70
  14. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  15. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  16. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  17. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  18. 2022Control Transformer: Robot Navigation in Unknown Environments through PRM-Guided Return-Conditioned Sequence Modelingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  3. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  4. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  5. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  6. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Makingcited by 26
  7. 2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
  8. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 14
  9. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  10. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  11. 2024Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusioncited by 11
  12. 2024Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?cited by 7
3 more
  1. 2024Solving Continual Offline Reinforcement Learning with Decision Transformercited by 6
  2. 2024Continual Task Learning through Adaptive Policy Self-Compositioncited by 1
  3. 2026Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shiftscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Merging
  2. 2024Octo: An Open-Source Generalist Robot Policy
  3. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learning
  4. 2025Toward Deployable Multi-Robot Collaboration via a Symbolically-Guided Decision Transformer
  5. 2025Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
  6. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learning
  7. 2025Much Ado About Noising: Dispelling the Myths of Generative Robotic Control
  8. 2025A Careful Examination of Large Behavior Models for Multitask Dexterous Manipulation