# Decoupled Training: Return of Frustratingly Easy Multi-Domain Learning

arXiv: https://arxiv.org/abs/2309.10302 · HTML: https://mlsanity.com/paper/2309.10302/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 30 of 30)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Deep Hashing Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1706.07522/) (2017)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Detecting and Correcting for Label Shift with Black Box Predictors](https://mlsanity.com/paper/1802.03916/) (2018)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [Learning Multi-Domain Convolutional Neural Networks for Visual Tracking](https://mlsanity.com/paper/1510.07945/) (2015)
- [Multi-Adversarial Domain Adaptation](https://mlsanity.com/paper/1809.02176/) (2018)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [Regularized Learning for Domain Adaptation under Label Shifts](https://mlsanity.com/paper/1903.09734/) (2019)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [One Model to Serve All: Star Topology Adaptive Recommender for Multi-Domain CTR Prediction](https://mlsanity.com/paper/2101.11427/) (2021)
- [Learning Deep Feature Representations with Domain Guided Dropout for Person Re-identification](https://mlsanity.com/paper/1604.07528/) (2016)
- [A Unified Perspective on Multi-Domain and Multi-Task Learning](https://mlsanity.com/paper/1412.7489/) (2014)
- [Sequential Scenario-Specific Meta Learner for Online Recommendation](https://mlsanity.com/paper/1906.00391/) (2019)
- [Multi-Domain Adversarial Learning](https://mlsanity.com/paper/1903.09239/) (2019)
- [Dual Attentive Sequential Learning for Cross-Domain Click-Through Rate Prediction](https://mlsanity.com/paper/2106.02768/) (2021)
- [Efficient Multi-Domain Network Learning by Covariance Normalization](https://mlsanity.com/paper/1906.10267/) (2019)
- [Dual Adversarial Co-Learning for Multi-Domain Text Classification](https://mlsanity.com/paper/1909.08203/) (2019)
- [Scenario-aware and Mutual-based approach for Multi-scenario Recommendation in E-Commerce](https://mlsanity.com/paper/2012.08952/) (2020)
- [Boosting Binary Masks for Multi-Domain Learning through Affine Transformations](https://mlsanity.com/paper/2103.13894/) (2021)

## Built on by (most cited first, 1 of 1)

- [Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models](https://mlsanity.com/paper/2403.02756/) (2024)

## Abstract (arXiv)

Multi-domain learning (MDL) aims to train a model with minimal average risk across multiple overlapping but non-identical domains. To tackle the challenges of dataset bias and domain domination, numerous MDL approaches have been proposed from the perspectives of seeking commonalities by aligning distributions to reduce domain gap or reserving differences by implementing domain-specific towers, gates, and even experts. MDL models are becoming more and more complex with sophisticated network architectures or loss functions, introducing extra parameters and enlarging computation costs. In this paper, we propose a frustratingly easy and hyperparameter-free multi-domain learning method named Decoupled Training (D-Train). D-Train is a tri-phase general-to-specific training strategy that first pre-trains on all domains to warm up a root model, then post-trains on each domain by splitting into multi-heads, and finally fine-tunes the heads by fixing the backbone, enabling decouple training to achieve domain independence. Despite its extraordinary simplicity and efficiency, D-Train performs remarkably well in extensive evaluations of various datasets from standard benchmarks to applications of satellite imagery and recommender systems.