# On the importance of pre-training data volume for compact language models

arXiv: https://arxiv.org/abs/2010.03813 · HTML: https://mlsanity.com/paper/2010.03813/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021)
- [On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model](https://mlsanity.com/paper/2204.13509/) (2022)
- [Trained on 100 million words and still in shape: BERT meets British National Corpus](https://mlsanity.com/paper/2303.09859/) (2023)
- [Match the Script, Adapt if Multilingual: Analyzing the Effect of Multilingual Pretraining on Cross-lingual Transferability](https://mlsanity.com/paper/2203.10753/) (2022)
- [On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets](https://mlsanity.com/paper/2109.03537/) (2021)
- [How much pretraining data do language models need to learn syntax?](https://mlsanity.com/paper/2109.03160/) (2021)
- [MicroBERT: Effective Training of Low-resource Monolingual BERTs through Parameter Reduction and Multitask Learning](https://mlsanity.com/paper/2212.12510/) (2022)
- [Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?](https://mlsanity.com/paper/2110.13658/) (2021)
- [Feature-Level Debiased Natural Language Understanding](https://mlsanity.com/paper/2212.05421/) (2022)
- [Data-Efficient French Language Modeling with CamemBERTa](https://mlsanity.com/paper/2306.01497/) (2023)
- [FQuAD2.0: French Question Answering and knowing that you know nothing](https://mlsanity.com/paper/2109.13209/) (2021)
- [Legal Transformer Models May Not Always Help](https://mlsanity.com/paper/2109.06862/) (2021)
- [Benchmarking Transformers-based models on French Spoken Language Understanding tasks](https://mlsanity.com/paper/2207.09152/) (2022)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [TAROT: A Hierarchical Framework with Multitask Co-Pretraining on Semi-Structured Data towards Effective Person-Job Fit](https://mlsanity.com/paper/2401.07525/) (2024)
- [A First Context-Free Grammar Applied to Nawatl Corpora Augmentation](https://mlsanity.com/paper/2510.04945/) (2025)
- [Two CFG Nahuatl for automatic corpora expansion](https://mlsanity.com/paper/2512.14239/) (2025)
- [Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings](https://mlsanity.com/paper/2310.19420/) (2023)
- [Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?](https://mlsanity.com/paper/2311.00268/) (2023)
- [Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility](https://mlsanity.com/paper/2505.16277/) (2025)
- [Corpora deduplication or duplication in Natural Language Processing of few resourced languages ? A case of study: The Mexico's Nahuatl](https://mlsanity.com/paper/2604.07015/) (2026)

## Abstract (arXiv)

Recent advances in language modeling have led to computationally intensive and resource-demanding state-of-the-art models. In an effort towards sustainable practices, we study the impact of pre-training data volume on compact language models. Multiple BERT-based models are trained on gradually increasing amounts of French text. Through fine-tuning on the French Question Answering Dataset (FQuAD), we observe that well-performing models are obtained with as little as 100 MB of text. In addition, we show that past critically low amounts of pre-training data, an intermediate pre-training step on the task-specific corpus does not yield substantial improvements.