# A Distributional Perspective on Word Learning in Neural Language Models

arXiv: https://arxiv.org/abs/2502.05892 · HTML: https://mlsanity.com/paper/2502.05892/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 61 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learning](https://mlsanity.com/paper/2005.07064/) (2020)
- [A standardized Project Gutenberg corpus for statistical analysis of natural language and quantitative linguistics](https://mlsanity.com/paper/1812.08092/) (2018)
- [Probabilistic Latent Semantic Analysis](https://mlsanity.com/paper/1301.6705/) (2013)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability](https://mlsanity.com/paper/2308.15419/) (2023)
- [[Call for Papers] The 2nd BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2404.06214/) (2024)
- [On the Role of Context in Reading Time Prediction](https://mlsanity.com/paper/2409.08160/) (2024)
- [Visual Grounding Helps Learn Word Meanings in Low-Data Regimes](https://mlsanity.com/paper/2310.13257/) (2023)
- [Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations](https://mlsanity.com/paper/2405.13828/) (2024)
- [Accumulated Gradient Normalization](https://mlsanity.com/paper/1710.02368/) (2017)
- [Learning the meanings of function words from grounded language using a visual question answering model](https://mlsanity.com/paper/2308.08628/) (2023)
- [Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling](https://mlsanity.com/paper/2403.14551/) (2024)
- [A Computational Acquisition Model for Multimodal Word Categorization](https://mlsanity.com/paper/2205.05974/) (2022)

## Built on by (most cited first, 4 of 4)

- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models](https://mlsanity.com/paper/2505.05970/) (2025)
- [Child-directed speech facilitates production, not comprehension, in BabyLMs](https://mlsanity.com/paper/2606.01045/) (2026)
- [On the use of foundation models in cognitive science](https://mlsanity.com/paper/2608.07812/) (2026)

## Abstract (arXiv)

Language models (LMs) are increasingly being studied as models of human language learners. Due to the nascency of the field, it is not well-established whether LMs exhibit similar learning dynamics to humans, and there are few direct comparisons between learning trajectories in humans and models. Word learning trajectories for children are relatively well-documented, and recent work has tried to extend these investigations to language models. However, there are no widely agreed-upon metrics for word learning in language models. We take a distributional approach to this problem, defining lexical knowledge in terms of properties of the learned distribution for a target word. We argue that distributional signatures studied in prior work fail to capture key distributional information. Thus, we propose an array of signatures that improve on earlier approaches by capturing knowledge of both where the target word can and cannot occur as well as gradient preferences about the word's appropriateness. We obtain learning trajectories for a selection of small language models we train from scratch, study the relationship between different distributional signatures, compare how well they align with human word learning trajectories and interpretable lexical features, and address basic methodological questions about estimating these distributional signatures. Our metrics largely capture complementary information, suggesting that it is important not to rely on a single metric. However, across all metrics, language models' learning trajectories fail to correlate with those of children.