# PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models

arXiv: https://arxiv.org/abs/2006.09075 · HTML: https://mlsanity.com/paper/2006.09075/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 30 of 33)

- [PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains](https://mlsanity.com/paper/2102.12206/) (2021)
- [Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulations](https://mlsanity.com/paper/2204.02683/) (2022)
- [DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptation](https://mlsanity.com/paper/2202.12350/) (2022)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)
- [Improving Temporal Generalization of Pre-trained Language Models with Lexical Semantic Change](https://mlsanity.com/paper/2210.17127/) (2022)
- [Example-based Hypernetworks for Out-of-Distribution Generalization](https://mlsanity.com/paper/2203.14276/) (2022)
- [Unsupervised Domain Adaptation with Adapter](https://mlsanity.com/paper/2111.00667/) (2021)
- [DILBERT: Customized Pre-Training for Domain Adaptation withCategory Shift, with an Application to Aspect Extraction](https://mlsanity.com/paper/2109.00571/) (2021)
- [Measuring the Robustness of NLP Models to Domain Shifts](https://mlsanity.com/paper/2306.00168/) (2023)
- [Representation Projection Invariance Mitigates Representation Collapse](https://mlsanity.com/paper/2205.11603/) (2022)
- [Why Is It Hate Speech? Masked Rationale Prediction for Explainable Hate Speech Detection](https://mlsanity.com/paper/2211.00243/) (2022)
- [Model Compression for Domain Adaptation through Causal Effect Estimation](https://mlsanity.com/paper/2101.07086/) (2021)
- [Target-Oriented Fine-tuning for Zero-Resource Named Entity Recognition](https://mlsanity.com/paper/2107.10523/) (2021)
- [Explainable Misinformation Detection Across Multiple Social Media Platforms](https://mlsanity.com/paper/2203.11724/) (2022)
- [How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation?](https://mlsanity.com/paper/2401.17514/) (2024)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [From Stance to Concern: Adaptation of Propositional Analysis to New Tasks and Domains](https://mlsanity.com/paper/2203.10659/) (2022)
- [IDANI: Inference-time Domain Adaptation via Neuron-level Interventions](https://mlsanity.com/paper/2206.00259/) (2022)
- [Domain Confused Contrastive Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2207.04564/) (2022)
- [Dynamically Refined Regularization for Improving Cross-corpora Hate Speech Detection](https://mlsanity.com/paper/2203.12536/) (2022)
- [Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning](https://mlsanity.com/paper/2211.11468/) (2022)
- [Source-Free Domain Adaptation for Question Answering with Masked Self-training](https://mlsanity.com/paper/2212.09563/) (2022)
- [A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification](https://mlsanity.com/paper/2304.09820/) (2023)
- [Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis](https://mlsanity.com/paper/2402.14536/) (2024)
- [Unsupervised Reinforcement Adaptation for Class-Imbalanced Text Classification](https://mlsanity.com/paper/2205.13139/) (2022)
- [Universal Domain Adaptation for Robust Handling of Distributional Shifts in NLP](https://mlsanity.com/paper/2310.14849/) (2023)
- [TACIT: A Target-Agnostic Feature Disentanglement Framework for Cross-Domain Text Classification](https://mlsanity.com/paper/2312.17263/) (2023)
- [Multi-Domain Explainability of Preferences](https://mlsanity.com/paper/2505.20088/) (2025)
- [Domain Adaptation from Scratch](https://mlsanity.com/paper/2209.00830/) (2022)
- [Domain Classification-based Source-specific Term Penalization for Domain Adaptation in Hate-speech Detection](https://mlsanity.com/paper/2209.08681/) (2022)

## Abstract (arXiv)

Pivot-based neural representation models have lead to significant progress in domain adaptation for NLP. However, previous works that follow this approach utilize only labeled data from the source domain and unlabeled data from the source and target domains, but neglect to incorporate massive unlabeled corpora that are not necessarily drawn from these domains. To alleviate this, we propose PERL: A representation learning model that extends contextualized word embedding models such as BERT with pivot-based fine-tuning. PERL outperforms strong baselines across 22 sentiment classification domain adaptation setups, improves in-domain model performance, yields effective reduced-size models and increases model stability.