mlsanityData to 7 Oct 2026

The Effects of In-domain Corpus Size on pre-training BERT

arXiv 2212.07914 ↗First posted 15 Dec 2022cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Many prior language modeling efforts have shown that pre-training on an in-domain corpus can significantly improve performance on downstream domain-specific NLP tasks. However, the difficulties associated with collecting enough in-domain data might discourage researchers from approaching this pre-training task. In this paper, we conducted a series of experiments by pre-training Bidirectional Encoder Representations from Transformers (BERT) with different sizes of biomedical corpora. The results demonstrate that pre-training on a relatively small amount of in-domain data (4GB) with limited training steps, can lead to better performance on downstream domain-specific NLP tasks compared with fine-tuning models pre-trained on general corpora.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 16 of its 26 references that are in the map. First posted on arXiv 15 Dec 2022.

Builds on

16 of 26 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2017Mixed Precision Trainingcited by 1,135
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  12. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
4 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  2. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  3. 2018ML-Net: multi-label classification of biomedical texts with deep neural networkscited by 17
  4. 2020Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Mediacited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 0; 2026: 2.

4 papers.

  1. 2024A Survey of Pre-trained Language Models for Processing Scientific Textcited by 7
  2. 2026Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimizationcited by 5
  3. 2023German FinBERT: A German Pre-trained Language Modelcited by 1
  4. 2026Towards Secure Cloud-Native Computing: Unveiling Kubernetes Misconfigurations with Large Language Modelscited by 0