# Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policies

arXiv: https://arxiv.org/abs/2303.07551 · HTML: https://mlsanity.com/paper/2303.07551/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 14 Mar 2023.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Learning Universal Policies via Text-Guided Video Generation](https://mlsanity.com/paper/2302.00111/) (2023)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Dataless Knowledge Fusion by Merging Weights of Language Models](https://mlsanity.com/paper/2212.09849/) (2022)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Towards Continual Reinforcement Learning: A Review and Perspectives](https://mlsanity.com/paper/2012.13490/) (2020)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models](https://mlsanity.com/paper/2208.03306/) (2022)
- [Patching open-vocabulary models by interpolating weights](https://mlsanity.com/paper/2208.05592/) (2022)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Fusing finetuned models for better pretraining](https://mlsanity.com/paper/2204.03044/) (2022)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models](https://mlsanity.com/paper/2302.07027/) (2023)
- [Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials](https://mlsanity.com/paper/2210.05178/) (2022)
- [Can Wikipedia Help Offline Reinforcement Learning?](https://mlsanity.com/paper/2201.12122/) (2022)
- [Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes](https://mlsanity.com/paper/2211.15144/) (2022)
- [Formal Algorithms for Transformers](https://mlsanity.com/paper/2207.09238/) (2022)
- [Re-basin via implicit Sinkhorn differentiation](https://mlsanity.com/paper/2212.12042/) (2022)

## Built on by (most cited first, 15 of 15)

- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [WARM: On the Benefits of Weight Averaged Reward Models](https://mlsanity.com/paper/2401.12187/) (2024)
- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [Merging Multi-Task Models via Weight-Ensembling Mixture of Experts](https://mlsanity.com/paper/2402.00433/) (2024)
- [WARP: On the Benefits of Weight Averaged Rewarded Policies](https://mlsanity.com/paper/2406.16768/) (2024)
- [Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making](https://mlsanity.com/paper/2310.03022/) (2023)
- [Cultural Palette: Pluralising Culture Alignment via Multi-agent Palette](https://mlsanity.com/paper/2412.11167/) (2024)
- [Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspace](https://mlsanity.com/paper/2410.13910/) (2024)
- [Skill Expansion and Composition in Parameter Space](https://mlsanity.com/paper/2502.05932/) (2025)
- [Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusion](https://mlsanity.com/paper/2311.07682/) (2023)
- [Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion](https://mlsanity.com/paper/2406.09770/) (2024)
- [Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?](https://mlsanity.com/paper/2405.12094/) (2024)
- [Solving Continual Offline Reinforcement Learning with Decision Transformer](https://mlsanity.com/paper/2401.08478/) (2024)
- [Continual Task Learning through Adaptive Policy Self-Composition](https://mlsanity.com/paper/2411.11364/) (2024)
- [Domain Arithmetic: One-Shot VLA Adaptation under Environmental Shifts](https://mlsanity.com/paper/2607.00666/) (2026)

## Abstract (arXiv)

Recent work has shown the promise of creating generalist, transformer-based, models for language, vision, and sequential decision-making problems. To create such models, we generally require centralized training objectives, data, and compute. It is of interest if we can more flexibly create generalist policies by merging together multiple, task-specific, individually trained policies. In this work, we take a preliminary step in this direction through merging, or averaging, subsets of Decision Transformers in parameter space trained on different MuJoCo locomotion problems, forming multi-task models without centralized training. We also demonstrate the importance of various methodological choices when merging policies, such as utilizing common pre-trained initializations, increasing model capacity, and utilizing Fisher information for weighting parameter importance. In general, we believe research in this direction could help democratize and distribute the process that forms multi-task robotics policies. Our implementation is available at https://github.com/daniellawson9999/merging-decision-transformers.