[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus
Not on the map. Why?
Abstract (arXiv)
After last year's successful BabyLM Challenge, the competition will be hosted again in 2024/2025. The overarching goals of the challenge remain the same; however, some of the competition rules will be different. The big changes for this year's competition are as follows: First, we replace the loose track with a paper track, which allows (for example) non-model-based submissions, novel cognitively-inspired benchmarks, or analysis techniques. Second, we are relaxing the rules around pretraining data, and will now allow participants to construct their own datasets provided they stay within the 100M-word or 10M-word budget. Third, we introduce a multimodal vision-and-language track, and will release a corpus of 50% text-only and 50% image-text multimodal data as a starting point for LM model training. The purpose of this CfP is to provide rules for this year's challenge, explain these rule changes and their rationale in greater detail, give a timeline of this year's competition, and provide answers to frequently asked questions from last year's challenge.
Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 7 of its 16 references that are in the map. First posted on arXiv 9 Apr 2024.
Builds on
7 of 16 references.
References · 7
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2018Datasheets for Datasetscited by 1,371
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
- 2019Connecting Vision and Language with Localized Narrativescited by 225
- 2018A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguisticscited by 64
- 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
- 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
Built on by
Citing papers per year: 2024: 10; 2025: 14; 2026: 2.
26 papers.
Sort:
By area:
- 2024GPT or BERT: why not both?cited by 26
- 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24
- 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
- 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
- 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshopcited by 10
- 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
- 2025Subword models struggle with word learning, but surprisal hides itcited by 6
- 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learningcited by 6
- 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Datacited by 5
- 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamascited by 5
- 2024What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performancecited by 4
- 2024BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiencycited by 4
14 more
- 2025A Distributional Perspective on Word Learning in Neural Language Modelscited by 4
- 2025Pretraining Language Models for Diachronic Linguistic Change Discoverycited by 4
- 2024Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Trainingcited by 3
- 2025UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactionscited by 3
- 2025LongTail-Swap: benchmarking language models' abilities on rare wordscited by 3
- 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-trainingcited by 2
- 2025Influence-driven Curriculum Learning for Pre-training on Limited Datacited by 2
- 2025Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Modelscited by 1
- 2024Choosy Babies Need One Coach: Inducing Mode-Seeking Behavior in BabyLlama with Reverse KL Divergencecited by 0
- 2024When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?cited by 0
- 2025Syntactic Learnability of Echo State Neural Language Models at Scalecited by 0
- 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibilitycited by 0
- 2025PaPaformer: Language Model from Pre-trained Parallel Pathscited by 0
- 2026A retrospective analysis on the use of LLMs to study infant syntax learningcited by 0
- 2026A retrospective analysis on the use of LLMs to study infant syntax learning22 Sep 2026
- 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop23 Feb 2026
- 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models6 Oct 2025
- 2025LongTail-Swap: benchmarking language models' abilities on rare words5 Oct 2025
- 2025Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models2 Oct 2025
- 2025Influence-driven Curriculum Learning for Pre-training on Limited Data21 Aug 2025
- 2025PaPaformer: Language Model from Pre-trained Parallel Paths1 Aug 2025
- 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility22 May 2025
- 2025UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions28 Apr 2025
- 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learning13 Apr 2025
- 2025Pretraining Language Models for Diachronic Linguistic Change Discovery7 Apr 2025
- 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?14 Mar 2025
14 more
- 2025Syntactic Learnability of Echo State Neural Language Models at Scale3 Mar 2025
- 2025Subword models struggle with word learning, but surprisal hides it18 Feb 2025
- 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshop15 Feb 2025
- 2025A Distributional Perspective on Word Learning in Neural Language Models9 Feb 2025
- 2024AntLM: Bridging Causal and Masked Language Models4 Dec 2024
- 2024When Babies Teach Babies: Can student knowledge sharing outperform Teacher-Guided Distillation on small datasets?25 Nov 2024
- 2024BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiency14 Nov 2024
- 2024What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performance11 Nov 2024
- 2024GPT or BERT: why not both?31 Oct 2024
- 2024Choosy Babies Need One Coach: Inducing Mode-Seeking Behavior in BabyLlama with Reverse KL Divergence29 Oct 2024
- 2024Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training20 Oct 2024
- 2024BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training20 Oct 2024
- 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas2 Oct 2024
- 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Data25 Sep 2024
- 2024GPT or BERT: why not both?cited by 26
- 2025BabyLM Turns 3: Call for papers for the 2025 BabyLM workshopcited by 24
- 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
- 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Modelscited by 11
- 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshopcited by 10
- 2024AntLM: Bridging Causal and Masked Language Modelscited by 8
- 2025Subword models struggle with word learning, but surprisal hides itcited by 6
- 2024BabyLlama-2: Ensemble-Distilled Models Consistently Outperform Teachers With Limited Datacited by 5
- 2024Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamascited by 5
- 2024What Should Baby Models Read? Exploring Sample-Efficient Data Composition on Model Performancecited by 4
- 2024BabyLM Challenge: Exploring the Effect of Variation Sets on Language Model Training Efficiencycited by 4
- 2025A Distributional Perspective on Word Learning in Neural Language Modelscited by 4
12 most cited of 25 All 25 (JSON)
- 2026A retrospective analysis on the use of LLMs to study infant syntax learning22 Sep 2026
- 2026BabyLM Turns 4 and Goes Multilingual: Call for Papers for the 2026 BabyLM Workshop23 Feb 2026
- 2025Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models6 Oct 2025
- 2025LongTail-Swap: benchmarking language models' abilities on rare words5 Oct 2025
- 2025Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models2 Oct 2025
- 2025Influence-driven Curriculum Learning for Pre-training on Limited Data21 Aug 2025
- 2025PaPaformer: Language Model from Pre-trained Parallel Paths1 Aug 2025
- 2025Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility22 May 2025
- 2025UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions28 Apr 2025
- 2025Pretraining Language Models for Diachronic Linguistic Change Discovery7 Apr 2025
- 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?14 Mar 2025
- 2025Syntactic Learnability of Echo State Neural Language Models at Scale3 Mar 2025
12 newest of 25 All 25 (JSON)