mlsanityData to 7 Oct 2026

Pre-training LLMs using human-like development data corpus

arXiv 2311.04666 ↗First posted 8 Nov 2023cs.CL cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Pre-trained Large Language Models (LLMs) have shown success in a diverse set of language inference and understanding tasks. The pre-training stage of LLMs looks at a large corpus of raw textual data. The BabyLM shared task compares LLM pre-training to human language acquisition, where the number of tokens seen by 13-year-old kids is magnitudes smaller than the number of tokens seen by LLMs. In this work, we pre-train and evaluate LLMs on their ability to learn contextual word representations using roughly the same number of tokens as seen by children. We provide a strong set of baselines; with different architectures, evaluation of changes in performance across epochs, and reported pre-training metrics for the strict small and strict tracks of the task. We also try to loosely replicate the RoBERTa baseline given by the task organizers to observe the training robustness to hyperparameter selection and replicability. We provide the submission details to the strict and strict-small tracks in this report.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

18 of 35 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  11. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 206
  12. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
6 more
  1. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  2. 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
  3. 2018Improving Question Answering by Commonsense-Based Pre-Trainingcited by 66
  4. 2021How much pretraining data do language models need to learn syntax?cited by 21
  5. 2023Imitation versus Innovation: What children can do that large language and language-and-vision models cannot (yet)?cited by 13
  6. 2022LogiGAN: Learning Logical Reasoning via Adversarial Pre-trainingcited by 11

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  2. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  3. 2024Incremental Comprehension of Garden-Path Sentences by Large Language Models: Semantic Interpretation, Syntactic Re-Analysis, and Attentioncited by 10
  4. 2025The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theoriescited by 2
  5. 2025Context-Aware Toxicity Detection in Multiplayer Games: Integrating Domain-Adaptive Pretraining and Match Metadatacited by 0