# Is Child-Directed Speech Effective Training Data for Language Models?

arXiv: https://arxiv.org/abs/2408.03617 · HTML: https://mlsanity.com/paper/2408.03617/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 51 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 15 of 15)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [When Do You Need Billions of Words of Pretraining Data?](https://mlsanity.com/paper/2011.04946/) (2020)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [CLIMB: Curriculum Learning for Infant-inspired Model Building](https://mlsanity.com/paper/2311.08886/) (2023)
- [Visual Grounding Helps Learn Word Meanings in Low-Data Regimes](https://mlsanity.com/paper/2310.13257/) (2023)
- [A systematic investigation of learnability from single child linguistic input](https://mlsanity.com/paper/2402.07899/) (2024)
- [Emergent Abilities in Reduced-Scale Generative Language Models](https://mlsanity.com/paper/2404.02204/) (2024)
- [Infant directed speech is consistent with teaching](https://mlsanity.com/paper/1606.01175/) (2016)
- [Can training neural language models on a curriculum with developmentally plausible data improve alignment with human reading behavior?](https://mlsanity.com/paper/2311.18761/) (2023)

## Built on by (most cited first, 24 of 24)

- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining](https://mlsanity.com/paper/2412.15285/) (2024)
- [Both Direct and Indirect Evidence Contribute to Dative Alternation Preferences in Language Models](https://mlsanity.com/paper/2503.20850/) (2025)
- [Do Construction Distributions Shape Formal Language Learning In German BabyLMs?](https://mlsanity.com/paper/2503.11593/) (2025)
- [Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)](https://mlsanity.com/paper/2510.20358/) (2025)
- [Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition](https://mlsanity.com/paper/2502.04795/) (2025)
- [BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data](https://mlsanity.com/paper/2510.10159/) (2025)
- [Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models](https://mlsanity.com/paper/2505.23689/) (2025)
- [Teacher Demonstrations in a BabyLM's Zone of Proximal Development for Contingent Multi-Turn Interaction](https://mlsanity.com/paper/2510.20411/) (2025)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models](https://mlsanity.com/paper/2603.29552/) (2026)
- [Scaling LLM Pre-training with Vocabulary Curriculum](https://mlsanity.com/paper/2502.17910/) (2025)
- [UD-English-CHILDES: A Collected Resource of Gold and Silver Universal Dependencies Trees for Child Language Interactions](https://mlsanity.com/paper/2504.20304/) (2025)
- [HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models](https://mlsanity.com/paper/2605.19341/) (2026)
- [BabyReasoningBench: Generating Developmentally-Inspired Reasoning Tasks for Evaluating Baby Language Models](https://mlsanity.com/paper/2601.18933/) (2026)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [A Unified Assessment of the Poverty of the Stimulus Argument for Neural Language Models](https://mlsanity.com/paper/2602.09992/) (2026)
- [To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval](https://mlsanity.com/paper/2604.00715/) (2026)
- [Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility](https://mlsanity.com/paper/2505.16277/) (2025)
- [BabyLM's First Constructions: Causal probing provides a signal of learning](https://mlsanity.com/paper/2506.02147/) (2025)
- [Do Syntactic Categories Help in Developmentally Motivated Curriculum Learning for Language Models?](https://mlsanity.com/paper/2511.08199/) (2025)
- [Beyond Length: Context-Aware Expansion and Independence as Developmentally Sensitive Evaluation in Child Utterances](https://mlsanity.com/paper/2602.05392/) (2026)
- [Is Child-Directed Language Optimized for Word Learning? A Computational Study of Verb Meaning Acquisition](https://mlsanity.com/paper/2605.12047/) (2026)
- [Children, but not language models, show accelerating returns in word learning](https://mlsanity.com/paper/2608.17120/) (2026)

## Abstract (arXiv)

While high-performing language models are typically trained on hundreds of billions of words, human children become fluent language users with a much smaller amount of data. What are the features of the data they receive, and how do these features support language modeling objectives? To investigate this question, we train GPT-2 and RoBERTa models on 29M words of English child-directed speech and a new matched, synthetic dataset (TinyDialogues), comparing to OpenSubtitles, Wikipedia, and a heterogeneous blend of datasets from the BabyLM challenge. We evaluate the syntactic and semantic knowledge of these models using developmentally-inspired evaluations. Through pretraining experiments, we test whether the global developmental ordering or the local discourse ordering of children's training data supports high performance relative to other datasets. The local properties of the data affect model results, but surprisingly, global properties do not. Further, child language input is not uniquely valuable for training language models. These findings support the hypothesis that, rather than proceeding from better data, the child's learning algorithm is substantially more data-efficient than current language modeling techniques.