mlsanityData to 7 Oct 2026

Too Much Information: Keeping Training Simple for BabyLMs

arXiv 2311.01955 ↗First posted 3 Nov 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper details the work of the University of Groningen for the BabyLM Challenge. We follow the idea that, like babies, language models should be introduced to simpler concepts first and build off of that knowledge to understand more complex concepts. We examine this strategy of simple-then-complex through a variety of lenses, namely context size, vocabulary, and overall linguistic complexity of the data. We find that only one, context size, is truly beneficial to training a language model. However this simple change to context size gives us improvements of 2 points on average on (Super)GLUE tasks, 1 point on MSGS tasks, and 12% on average on BLiMP tasks. Our context-limited model outperforms the baseline that was trained on 10× the amount of data.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 3 Nov 2023.

Builds on

21 of 28 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  11. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  12. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
9 more
  1. 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 201
  2. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  3. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  4. 2017Curriculum Learning and Minibatch Bucketing in Neural Machine Translationcited by 63
  5. 2018An Empirical Exploration of Curriculum Learning for Neural Machine Translationcited by 58
  6. 2016Visualizing and Understanding Curriculum Learning for Long Short-Term Memory Networkscited by 44
  7. 2021Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learningcited by 34
  8. 2023Byte-Level Grammatical Error Correction Using Synthetic and Curated Corporacited by 7
  9. 2022The Importance of Context in Very Low Resource Language Modelingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  2. 2024Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Trainingcited by 3
  3. 2025Mask and You Shall Receive: Optimizing Masked Language Modeling For Pretraining BabyLMscited by 2
  4. 2024Are BabyLMs Second Language Learners?cited by 0
  5. 2024BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Modelscited by 0
  6. 2026Level Up: Defining and Exploiting Transitional Problems for Curriculum Learningcited by 0