# Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media

arXiv: https://arxiv.org/abs/2010.01150 · HTML: https://mlsanity.com/paper/2010.01150/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Discontinuous Named Entity Recognition as Maximal Clique Discovery](https://mlsanity.com/paper/2106.00218/) (2021)
- [Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks](https://mlsanity.com/paper/2311.01038/) (2023)
- [To Share or not to Share: Predicting Sets of Sources for Model Transfer Learning](https://mlsanity.com/paper/2104.08078/) (2021)
- [Automatic Document Selection for Efficient Encoder Pretraining](https://mlsanity.com/paper/2210.10951/) (2022)
- [Genre as Weak Supervision for Cross-lingual Dependency Parsing](https://mlsanity.com/paper/2109.04733/) (2021)
- [MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model](https://mlsanity.com/paper/2109.06605/) (2021)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [How Universal is Genre in Universal Dependencies?](https://mlsanity.com/paper/2112.04971/) (2021)
- [The Effects of In-domain Corpus Size on pre-training BERT](https://mlsanity.com/paper/2212.07914/) (2022)
- [Selecting Subsets of Source Data for Transfer Learning with Applications in Metal Additive Manufacturing](https://mlsanity.com/paper/2401.08715/) (2024)
- [Mixed-effects transformers for hierarchical adaptation](https://mlsanity.com/paper/2205.01749/) (2022)
- [Sort by Structure: Language Model Ranking as Dependency Probing](https://mlsanity.com/paper/2206.04935/) (2022)
- [Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain](https://mlsanity.com/paper/2403.10576/) (2024)
- [German FinBERT: A German Pre-trained Language Model](https://mlsanity.com/paper/2311.08793/) (2023)
- [Extraction of Medication Names from Twitter Using Augmentation and an Ensemble of Language Models](https://mlsanity.com/paper/2111.06664/) (2021)
- [S2F-NER: Exploring Sequence-to-Forest Generation for Complex Entity Recognition](https://mlsanity.com/paper/2310.18944/) (2023)

## Abstract (arXiv)

Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when models are pretrained on in-domain data. Often, the pretraining data used in these models are selected based on their subject matter, e.g., biology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0.