# [Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus

arXiv: https://arxiv.org/abs/2404.06214 · HTML: https://mlsanity.com/paper/2404.06214/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 16 references that are in the map.
- First posted on arXiv 9 Apr 2024.

## Builds on (most cited first, 7 of 7)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Connecting Vision and Language with Localized Narratives](https://mlsanity.com/paper/1912.03098/) (2019)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Trained on 100 million words and still in shape: BERT meets British National Corpus](https://mlsanity.com/paper/2303.09859/) (2023)

## Built on by (most cited first, 26 of 26)

- [GPT or BERT: why not both?](https://mlsanity.com/paper/2410.24159/) (2024)
- [BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop](https://mlsanity.com/paper/2502.10645/) (2025)
- [BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop](https://mlsanity.com/paper/2602.20092/) (2026)
- [Do Construction Distributions Shape Formal Language Learning In German BabyLMs?](https://mlsanity.com/paper/2503.11593/) (2025)
- [Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models](https://mlsanity.com/paper/2510.04764/) (2025)
- [AntLM: Bridging Causal and Masked Language Models](https://mlsanity.com/paper/2412.03275/) (2024)
- [Subword models struggle with word learning, but surprisal hides it](https://mlsanity.com/paper/2502.12835/) (2025)
- [Pretraining Language Models for Diachronic Linguistic Change Discovery](https://mlsanity.com/paper/2504.05523/) (2025)
- [BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learning](https://mlsanity.com/paper/2504.09426/) (2025)
- [BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data](https://mlsanity.com/paper/2409.17312/) (2024)
- [Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas](https://mlsanity.com/paper/2410.01487/) (2024)
- [What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance](https://mlsanity.com/paper/2411.06672/) (2024)
- [BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency](https://mlsanity.com/paper/2411.09587/) (2024)
- [A Distributional Perspective on Word Learning in Neural Language Models](https://mlsanity.com/paper/2502.05892/) (2025)
- [BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training](https://mlsanity.com/paper/2410.15365/) (2024)
- [Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training](https://mlsanity.com/paper/2410.15509/) (2024)
- [UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions](https://mlsanity.com/paper/2504.20304/) (2025)
- [LongTail-Swap: benchmarking language models' abilities on rare words](https://mlsanity.com/paper/2510.04268/) (2025)
- [Influence-driven Curriculum Learning for Pre-training on Limited Data](https://mlsanity.com/paper/2508.15475/) (2025)
- [Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models](https://mlsanity.com/paper/2510.01845/) (2025)
- [Choosy Babies Need One Coach: Inducing Mode-Seeking Behavior in BabyLlama with Reverse KL Divergence](https://mlsanity.com/paper/2410.22081/) (2024)
- [When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?](https://mlsanity.com/paper/2411.16487/) (2024)
- [Syntactic Learnability of Echo State Neural Language Models at Scale](https://mlsanity.com/paper/2503.01724/) (2025)
- [Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility](https://mlsanity.com/paper/2505.16277/) (2025)
- [PaPaformer: Language Model from Pre-trained Parallel Paths](https://mlsanity.com/paper/2508.00544/) (2025)
- [A retrospective analysis on the use of LLMs to study infant syntax learning](https://mlsanity.com/paper/2609.26539/) (2026)

## Abstract (arXiv)

After last year's successful BabyLM Challenge, the competition will be hosted again in 2024/2025. The overarching goals of the challenge remain the same; however, some of the competition rules will be different. The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques. Second, we are relaxing the rules around pretraining data, and will now allow participants to construct their own datasets provided they stay within the 100M-word or 10M-word budget. Third, we introduce a multimodal vision-and-language track, and will release a corpus of 50% text-only and 50% image-text multimodal data as a starting point for LM model training. The purpose of this CfP is to provide rules for this year's challenge, explain these rule changes and their rationale in greater detail, give a timeline of this year's competition, and provide answers to frequently asked questions from last year's challenge.