mlsanityData to 7 Oct 2026

Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media

arXiv 2010.01150 ↗First posted 2 Oct 2020cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Recent studies on domain-specific BERT models show that effectiveness on downstream tasks can be improved when models are pretrained on in-domain data. Often, the pretraining data used in these models are selected based on their subject matter, e.g., biology or computer science. Given the range of applications using social media text, and its unique language variety, we pretrain two models on tweets and forum text respectively, and empirically demonstrate the effectiveness of these two resources. In addition, we investigate how similarity measures can be used to nominate in-domain pretraining data. We publicly release our pretrained models at https://bit.ly/35RpTf0.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 4; 2023: 3; 2024: 2.

16 papers.

Sort:

  1. 2021Discontinuous Named Entity Recognition as Maximal Clique Discoverycited by 17
  2. 2023Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networkscited by 14
  3. 2021To Share or not to Share: Predicting Sets of Sources for Model Transfer Learningcited by 11
  4. 2022Automatic Document Selection for Efficient Encoder Pretrainingcited by 11
  5. 2021Genre as Weak Supervision for Cross-lingual Dependency Parsingcited by 8
  6. 2021MDAPT: Multilingual Domain Adaptive Pretraining in a Single Modelcited by 6
  7. 2021On the Universality of Deep Contextual Language Modelscited by 4
  8. 2021How Universal is Genre in Universal Dependencies?cited by 4
  9. 2022The Effects of In-domain Corpus Size on pre-training BERTcited by 4
  10. 2024Selecting Subsets of Source Data for Transfer Learning with Applications in Metal Additive Manufacturingcited by 3
  11. 2022Mixed-effects transformers for hierarchical adaptationcited by 2
  12. 2022Sort by Structure: Language Model Ranking as Dependency Probingcited by 2
4 more
  1. 2024Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domaincited by 2
  2. 2023German FinBERT: A German Pre-trained Language Modelcited by 1
  3. 2021Extraction of Medication Names from Twitter Using Augmentation and an Ensemble of Language Modelscited by 0
  4. 2023S2F-NER: Exploring Sequence-to-Forest Generation for Complex Entity Recognitioncited by 0