# Transformer Based Multi-Source Domain Adaptation

arXiv: https://arxiv.org/abs/2009.07806 · HTML: https://mlsanity.com/paper/2009.07806/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- First posted on arXiv 16 Sep 2020.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Built on by (most cited first, 23 of 23)

- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [Domain Adaptation for Time Series Forecasting via Attention Sharing](https://mlsanity.com/paper/2102.06828/) (2021)
- [PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains](https://mlsanity.com/paper/2102.12206/) (2021)
- [DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptation](https://mlsanity.com/paper/2202.12350/) (2022)
- [Cross-Domain Label-Adaptive Stance Detection](https://mlsanity.com/paper/2104.07467/) (2021)
- [Pre-train or Annotate? Domain Adaptation with a Constrained Budget](https://mlsanity.com/paper/2109.04711/) (2021)
- [On the Domain Adaptation and Generalization of Pretrained Language Models: A Survey](https://mlsanity.com/paper/2211.03154/) (2022)
- [CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understanding](https://mlsanity.com/paper/2105.10912/) (2021)
- [Conditional Adversarial Networks for Multi-Domain Text Classification](https://mlsanity.com/paper/2102.10176/) (2021)
- [Example-based Hypernetworks for Out-of-Distribution Generalization](https://mlsanity.com/paper/2203.14276/) (2022)
- [Energy-Based Test Sample Adaptation for Domain Generalization](https://mlsanity.com/paper/2302.11215/) (2023)
- [DAME: Domain Adaptation for Matching Entities](https://mlsanity.com/paper/2204.09244/) (2022)
- [Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks](https://mlsanity.com/paper/2111.01340/) (2021)
- [Determining the Credibility of Science Communication](https://mlsanity.com/paper/2105.14473/) (2021)
- [Multi-View Active Learning for Short Text Classification in User-Generated Data](https://mlsanity.com/paper/2112.02611/) (2021)
- [Multiple-Source Domain Adaptation via Coordinated Domain Encoders and Paired Classifiers](https://mlsanity.com/paper/2201.11870/) (2022)
- [Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis](https://mlsanity.com/paper/2402.14536/) (2024)
- [SPARTAN: Sparse Hierarchical Memory for Parameter-Efficient Transformers](https://mlsanity.com/paper/2211.16634/) (2022)
- [Classical Sequence Match is a Competitive Few-Shot One-Class Learner](https://mlsanity.com/paper/2209.06394/) (2022)
- [Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sources](https://mlsanity.com/paper/2311.09732/) (2023)
- [Learning to Generalize Unseen Domains via Multi-Source Meta Learning for Text Classification](https://mlsanity.com/paper/2409.13787/) (2024)
- [Similarity-Based Domain Adaptation with LLMs](https://mlsanity.com/paper/2503.05281/) (2025)

## Abstract (arXiv)

In practical machine learning settings, the data on which a model must make predictions often come from a different distribution than the data it was trained on. Here, we investigate the problem of unsupervised multi-source domain adaptation, where a model is trained on labelled data from multiple source domains and must make predictions on a domain for which no labelled data has been seen. Prior work with CNNs and RNNs has demonstrated the benefit of mixture of experts, where the predictions of multiple domain expert classifiers are combined; as well as domain adversarial training, to induce a domain agnostic representation space. Inspired by this, we investigate how such methods can be effectively applied to large pretrained transformer models. We find that domain adversarial training has an effect on the learned representations of these models while having little effect on their performance, suggesting that large transformer-based models are already relatively robust across domains. Additionally, we show that mixture of experts leads to significant performance improvements by comparing several variants of mixing functions, including one novel mixture based on attention. Finally, we demonstrate that the predictions of large pretrained transformer based domain experts are highly homogenous, making it challenging to learn effective functions for mixing their predictions.