mlsanityData to 7 Oct 2026

Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

arXiv 2301.11796 ↗First posted 27 Jan 2023cs.CLcited by 62

Not on the map. Why?

Abstract (arXiv)

We present the call for papers for the BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus. This shared task is intended for participants with an interest in small scale language modeling, human language acquisition, low-resource NLP, and cognitive modeling. In partnership with CoNLL and CMCL, we provide a platform for approaches to pretraining with a limited-size corpus sourced from data inspired by the input to children. The task has three tracks, two of which restrict the training data to pre-released datasets of 10M and 100M words and are dedicated to explorations of approaches such as architectural variations, self-supervised objectives, or curriculum learning. The final track only restricts the amount of text used, allowing innovation in the choice of the data, its domain, and even its modality (i.e., data from sources other than text is welcome). We will release a shared evaluation pipeline which scores models on a variety of benchmarks and tasks, including targeted syntactic evaluations and natural language understanding.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Builds on 11 of its 20 references that are in the map. First posted on arXiv 27 Jan 2023.

Builds on

11 of 20 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  8. 2015The Goldilocks Principle: Reading Children's Books with Explicit Memory Representationscited by 108
  9. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  10. 2016Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learnercited by 68
  11. 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 16; 2025: 20; 2026: 8.

62 papers.

Sort:

By area:

  1. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  2. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  4. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  5. 2023Efficient Benchmarking of Language Modelscited by 71
  6. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  7. 2023Injecting structural hints: Using language models to study inductive biases in language learningcited by 33
  8. 2024Mission: Impossible Language Modelscited by 31
  9. 2023BabySLM: language-acquisition-friendly benchmark of self-supervised spoken language modelscited by 29
  10. 2024[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 26
  11. 2025Pre-training under infinite computecited by 24
  12. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
38 more
  1. 2024BudgetMLAgent: A Cost-Effective LLM Multi-Agent system for Automating Machine Learning Taskscited by 19
  2. 2023Visual Grounding Helps Learn Word Meanings in Low-Data Regimescited by 16
  3. 2025Position: The Most Expensive Part of an LLM should be its Training Datacited by 15
  4. 2024Limits of Transformer Language Models on Learning to Compose Algorithmscited by 13
  5. 2024Emergent Word Order Universals from Cognitively-Motivated Language Modelscited by 11
  6. 2024Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decodingcited by 11
  7. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  8. 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
  9. 2023ChatGPT in the Age of Generative AI and Large Language Models: A Concise Surveycited by 10
  10. 2023Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Modelscited by 9
  11. 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
  12. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  13. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  14. 2026Data-efficient pre-training by scaling synthetic megadocscited by 7
  15. 2025IPA-CHILDES & G2P+: Feature-Rich Resources for Cross-Lingual Phonology and Phonemic Language Modelingcited by 6
  16. 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learningcited by 6
  17. 2023On the Unexpected Abilities of Large Language Modelscited by 5
  18. 2024Gradient-based inference of abstract task representations for generalization in neural networkscited by 4
  19. 2025Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Modelscited by 4
  20. 2025Scaling LLM Pre-training with Vocabulary Curriculumcited by 3
  21. 2025LongTail-Swap: benchmarking language models' abilities on rare wordscited by 3
  22. 2023WhisBERT: Multimodal Text-Audio Language Modeling on 100M Wordscited by 2
  23. 2024Investigating grammatical abstraction in language models using few-shot learning of novel noun gendercited by 2
  24. 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 2
  25. 2024Building, Reusing, and Generalizing Abstract Representations from Concrete Sequencescited by 2
  26. 2025BabyLM's First Words: Word Segmentation as a Phonological Probing Taskcited by 2
  27. 2025Playpen: An Environment for Exploring Learning Through Conversational Interactioncited by 2
  28. 2025Influence-driven Curriculum Learning for Pre-training on Limited Datacited by 2
  29. 2025Masked Diffusion Language Models with Frequency-Informed Trainingcited by 2
  30. 2025Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modellingcited by 2
  31. 2026Zero-shot World Models Are Developmentally Efficient Learnerscited by 2
  32. 2023ToddlerBERTa: Exploiting BabyBERTa for Grammar Learning and Language Understandingcited by 1
  33. 2025Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performancecited by 1
  34. 2023Reducing the Need for Backpropagation and Discovering Better Optima With Explicit Optimizations of Neural Networkscited by 0
  35. 2023Explicit Foundation Model Optimization with Self-Attentive Feed-Forward Neural Unitscited by 0
  36. 2023Bit Cipher -- A Simple yet Powerful Word Representation System that Integrates Efficiently with Language Modelscited by 0
  37. 2023DYAD: A Descriptive Yet Abjuring Density efficient approximation to linear neural network layerscited by 0
  38. 2024Tracking Universal Features Through Fine-Tuning and Model Mergingcited by 0

50 most cited of 62 All 62 (JSON)