mlsanityData to 7 Oct 2026

On the importance of pre-training data volume for compact language models

arXiv 2010.03813 ↗First posted 8 Oct 2020cs.CL cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. In an effort towards sustainable practices, we study the impact of pre-training data volume on compact language models. Multiple BERT-based models are trained on gradually increasing amounts of French text. Through fine-tuning on the French Question Answering Dataset (FQuAD), we observe that well-performing models are obtained with as little as 100 MB of text. In addition, we show that past critically low amounts of pre-training data, an intermediate pre-training step on the task-specific corpus does not yield substantial improvements.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 4; 2024: 1; 2025: 3; 2026: 2.

21 papers.

Sort:

By area:

  1. 2021The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Modelscited by 118
  2. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  3. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  4. 2022Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferabilitycited by 26
  5. 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
  6. 2021How much pretraining data do language models need to learn syntax?cited by 21
  7. 2022MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learningcited by 12
  8. 2021Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?cited by 11
  9. 2022Feature-Level Debiased Natural Language Understandingcited by 10
  10. 2023Data-Efficient French Language Modeling with CamemBERTacited by 8
  11. 2021FQuAD2.0: French Question Answering and knowing that you know nothingcited by 7
  12. 2021Legal Transformer Models May Not Always Helpcited by 6
9 more
  1. 2022Benchmarking Transformers-based models on French Spoken Language Understanding taskscited by 3
  2. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  3. 2024TAROT: A Hierarchical Framework with Multitask Co-Pretraining on Semi-Structured Data towards Effective Person-Job Fitcited by 1
  4. 2025A First Context-Free Grammar Applied to Nawatl Corpora Augmentationcited by 1
  5. 2025Two CFG Nahuatl for automatic corpora expansioncited by 1
  6. 2023Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settingscited by 0
  7. 2023Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?cited by 0
  8. 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibilitycited by 0
  9. 2026Corpora deduplication or duplication in Natural Language Processing of few resourced languages ? A case of study: The Mexico's Nahuatlcited by 0