# Domain Adaptation for Sparse-Data Settings: What Do We Gain by Not Using Bert?

arXiv: https://arxiv.org/abs/2203.16926 · HTML: https://mlsanity.com/paper/2203.16926/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 49 references that are in the map.
- First posted on arXiv 31 Mar 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Adaptive Semi-supervised Learning for Cross-domain Sentiment Classification](https://mlsanity.com/paper/1809.00530/) (2018)
- [Simplified Neural Unsupervised Domain Adaptation](https://mlsanity.com/paper/1905.09153/) (2019)
- [Embedding Transfer for Low-Resource Medical Named Entity Recognition: A Case Study on Patient Mobility](https://mlsanity.com/paper/1806.02814/) (2018)
- [A Simple Regularization-based Algorithm for Learning Cross-Domain Word Embeddings](https://mlsanity.com/paper/1902.00184/) (2019)
- [Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling](https://mlsanity.com/paper/1904.02817/) (2019)
- [Shallow Domain Adaptive Embeddings for Sentiment Analysis](https://mlsanity.com/paper/1908.06082/) (2019)
- [Updating Pre-trained Word Vectors and Text Classifiers using Monolingual Alignment](https://mlsanity.com/paper/1910.06241/) (2019)
- [Unsupervised Domain Adaptation with Feature Embeddings](https://mlsanity.com/paper/1412.4385/) (2014)
- [AWE-CM Vectors: Augmenting Word Embeddings with a Clinical Metathesaurus](https://mlsanity.com/paper/1712.01460/) (2017)
- [Unsupervised Domain Adaptation: A Multi-task Learning-based Method](https://mlsanity.com/paper/1803.09208/) (2018)

## Built on by (most cited first, 1 of 1)

- [An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification](https://mlsanity.com/paper/2409.03203/) (2024)

## Abstract (arXiv)

The practical success of much of NLP depends on the availability of training data. However, in real-world scenarios, training data is often scarce, not least because many application domains are restricted and specific. In this work, we compare different methods to handle this problem and provide guidelines for building NLP applications when there is only a small amount of labeled training data available for a specific domain. While transfer learning with pre-trained language models outperforms other methods across tasks, alternatives do not perform much worse while requiring much less computational effort, thus significantly reducing monetary and environmental cost. We examine the performance tradeoffs of several such alternatives, including models that can be trained up to 175K times faster and do not require a single GPU.