# Word Acquisition in Neural Language Models

arXiv: https://arxiv.org/abs/2110.02406 · HTML: https://mlsanity.com/paper/2110.02406/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 57 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Multi-Agent Cooperation and the Emergence of (Natural) Language](https://mlsanity.com/paper/1612.07182/) (2016)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Distributional Semantics and Linguistic Theory](https://mlsanity.com/paper/1905.01896/) (2019)
- [BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps](https://mlsanity.com/paper/2005.04625/) (2020)
- [PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World](https://mlsanity.com/paper/2106.00188/) (2021)
- [Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate](https://mlsanity.com/paper/1807.04783/) (2018)
- [Polysemy and brevity versus frequency in language](https://mlsanity.com/paper/1904.00812/) (2019)
- [What are the Goals of Distributional Semantics?](https://mlsanity.com/paper/2005.02982/) (2020)
- [Putting words in context: LSTM language models and lexical ambiguity](https://mlsanity.com/paper/1906.05149/) (2019)

## Built on by (most cited first, 30 of 41)

- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [LLM Circuit Analyses Are Consistent Across Training and Scale](https://mlsanity.com/paper/2407.10827/) (2024)
- [Confidence Regulation Neurons in Language Models](https://mlsanity.com/paper/2406.16254/) (2024)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability](https://mlsanity.com/paper/2308.15419/) (2023)
- [When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages](https://mlsanity.com/paper/2311.09205/) (2023)
- [Visual Grounding Helps Learn Word Meanings in Low-Data Regimes](https://mlsanity.com/paper/2310.13257/) (2023)
- [A Review of the Applications of Deep Learning-Based Emergent Communication](https://mlsanity.com/paper/2407.03302/) (2024)
- [DevBench: A multimodal developmental benchmark for language learning](https://mlsanity.com/paper/2406.10215/) (2024)
- [Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale](https://mlsanity.com/paper/2510.24963/) (2025)
- [Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory](https://mlsanity.com/paper/2405.08707/) (2024)
- [Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations](https://mlsanity.com/paper/2405.13828/) (2024)
- [World-to-Words: Grounded Open Vocabulary Acquisition through Fast Mapping in Vision-Language Models](https://mlsanity.com/paper/2306.08685/) (2023)
- [Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models](https://mlsanity.com/paper/2308.01684/) (2023)
- [Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models](https://mlsanity.com/paper/2305.12544/) (2023)
- [A Natural Bias for Language Generation Models](https://mlsanity.com/paper/2212.09686/) (2022)
- [The Mechanistic Emergence of Symbol Grounding in Language Models](https://mlsanity.com/paper/2510.13796/) (2025)
- [A Computational Acquisition Model for Multimodal Word Categorization](https://mlsanity.com/paper/2205.05974/) (2022)
- [SPE: Symmetrical Prompt Enhancement for Fact Probing](https://mlsanity.com/paper/2211.07078/) (2022)
- [Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models](https://mlsanity.com/paper/2504.15471/) (2025)
- [Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models](https://mlsanity.com/paper/2505.05970/) (2025)
- [Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts](https://mlsanity.com/paper/2505.13281/) (2025)
- [Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining](https://mlsanity.com/paper/2606.26050/) (2026)
- [Measure More, Question More: Experimental Studies on Transformer-based Language Models and Complement Coercion](https://mlsanity.com/paper/2212.10536/) (2022)
- [Language models as tools for investigating the distinction between possible and impossible natural languages](https://mlsanity.com/paper/2512.09394/) (2025)
- [Large language models have learned to use language](https://mlsanity.com/paper/2512.12447/) (2025)
- [Large language models are not about natural language](https://mlsanity.com/paper/2512.13441/) (2025)

## Abstract (arXiv)

We investigate how neural language models acquire individual words during training, extracting learning curves and ages of acquisition for over 600 words on the MacArthur-Bates Communicative Development Inventory (Fenson et al., 2007). Drawing on studies of word acquisition in children, we evaluate multiple predictors for words' ages of acquisition in LSTMs, BERT, and GPT-2. We find that the effects of concreteness, word length, and lexical class are pointedly different in children and language models, reinforcing the importance of interaction and sensorimotor experience in child language acquisition. Language models rely far more on word frequency than children, but like children, they exhibit slower learning of words in longer utterances. Interestingly, models follow consistent patterns during training for both unidirectional and bidirectional models, and for both LSTM and Transformer architectures. Models predict based on unigram token frequencies early in training, before transitioning loosely to bigram probabilities, eventually converging on more nuanced predictions. These results shed light on the role of distributional learning mechanisms in children, while also providing insights for more human-like language acquisition in language models.