mlsanityData to 14 Oct 2026

Trained on 100 million words and still in shape: BERT meets British National Corpus

arXiv 2303.09859 ↗First posted 17 Mar 2023cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well-balanced, and publicly available English text source -- the British National Corpus. We show that pre-training on this carefully curated corpus can reach better performance than the original BERT model. We argue that this type of corpora has great potential as a language modeling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evaluate several training objectives and model architectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 44 of its 84 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

44 of 84 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
32 more
  1. 2018Deep contextualized word representationscited by 2,290
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  7. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  10. 2018Neural Network Acceptability Judgmentscited by 958
  11. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  13. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  14. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  15. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  16. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  17. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  18. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  19. 2019CamemBERT: a Tasty French Language Modelcited by 368
  20. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  21. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
  22. 2020Understanding the Difficulty of Training Transformerscited by 210
  23. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 200
  24. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 185
  25. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  26. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  27. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  28. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  29. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  30. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  31. 2020On the importance of pre-training data volume for compact language modelscited by 21
  32. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 14; 2026: 6.

39 papers.

Sort:

By area:

  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  2. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  3. 2024GPT or BERT: why not both?cited by 27
  4. 2024[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 26
  5. 2023NorBench -- A Benchmark for Norwegian Language Modelscited by 23
  6. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  7. 2023Not all layers are equally as important: Every Layer Counts BERTcited by 21
  8. 2023CLIMB: Curriculum Learning for Infant-inspired Model Buildingcited by 20
  9. 2024Investigating Critical Period Effects in Language Acquisition through Neural Language Modelscited by 16
  10. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 16
  11. 2025Multilingual Pretraining for Pixel Language Modelscited by 12
  12. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
27 more
  1. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  2. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  3. 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Datacited by 5
  4. 2023Tokenization with Factorized Subword Encodingcited by 4
  5. 2024KyrgyzNLP: Challenges, Progress, and Futurecited by 4
  6. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  7. 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 3
  8. 2025skLEP: A Slovak General Language Understanding Benchmarkcited by 3
  9. 2025LongTail-Swap: benchmarking language models' abilities on rare wordscited by 3
  10. 2024More Room for Language: Investigating the Effect of Retrieval on Language Modelscited by 2
  11. 2025Masked Diffusion Language Models with Frequency-Informed Trainingcited by 2
  12. 2024Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situationscited by 1
  13. 2024Correspondence Analysis and PMI-Based Word Embeddings: A Comparative Studycited by 1
  14. 2025Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performancecited by 1
  15. 2026Learning from Child-Directed Speech in Two-Language Scenarios: A French-English Case Studycited by 1
  16. 2026DunbaaBERT: From Sacrifice to Semanticscited by 1
  17. 2023Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settingscited by 0
  18. 2024When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?cited by 0
  19. 2024A surprisal oracle for when every layer countscited by 0
  20. 2025BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Contextcited by 0
  21. 2025Named Entity Recognition in Contextcited by 0
  22. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
  23. 2025TiME: Tiny Monolingual Encoders for Efficient NLP Pipelinescited by 0
  24. 2026Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasingcited by 0
  25. 2026Targeted Syntactic Evaluation of Language Models on Georgian Case Alignmentcited by 0
  26. 2026Pretraining and Benchmarking Modern Encoders for Latviancited by 0
  27. 2026A retrospective analysis on the use of LLMs to study infant syntax learningcited by 0