# Evaluating Neural Language Models as Cognitive Models of Language Acquisition

arXiv: https://arxiv.org/abs/2310.20093 · HTML: https://mlsanity.com/paper/2310.20093/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 19 of 19)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models](https://mlsanity.com/paper/2110.07736/) (2021)
- [Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks](https://mlsanity.com/paper/2001.03632/) (2020)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT](https://mlsanity.com/paper/2101.11043/) (2021)
- [What if This Modified That? Syntactic Interventions via Counterfactual Embeddings](https://mlsanity.com/paper/2105.14002/) (2021)
- [Recurrent Neural Networks in Linguistic Theory: Revisiting Pinker and Prince (1988) and the Past Tense Debate](https://mlsanity.com/paper/1807.04783/) (2018)
- [Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets](https://mlsanity.com/paper/1704.07121/) (2017)

## Built on by (most cited first, 15 of 15)

- [What Can String Probability Tell Us About Grammaticality?](https://mlsanity.com/paper/2510.16227/) (2025)
- [What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length](https://mlsanity.com/paper/2411.02528/) (2024)
- [Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Models](https://mlsanity.com/paper/2505.23689/) (2025)
- [TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2506.13487/) (2025)
- [A systematic investigation of learnability from single child linguistic input](https://mlsanity.com/paper/2402.07899/) (2024)
- [Large Language Model probabilities cannot distinguish between possible and impossible language](https://mlsanity.com/paper/2509.15114/) (2025)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Test](https://mlsanity.com/paper/2510.12463/) (2025)
- [ERAS: Evaluating the Robustness of Chinese NLP Models to Morphological Garden Path Errors](https://mlsanity.com/paper/2410.13057/) (2024)
- [Measuring Form and Function in Language Models](https://mlsanity.com/paper/2605.28616/) (2026)
- [Linear representations of grammaticality in neural language models](https://mlsanity.com/paper/2607.15175/) (2026)
- [Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic](https://mlsanity.com/paper/2603.16406/) (2026)
- [Child-directed speech facilitates production, not comprehension, in BabyLMs](https://mlsanity.com/paper/2606.01045/) (2026)
- [Which Forms of Caregiver Feedback Support Grammar Learning? A Reinforcement-Learning Study of Child-Like Language Models](https://mlsanity.com/paper/2609.08576/) (2026)
- [Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality](https://mlsanity.com/paper/2609.34936/) (2026)

## Abstract (arXiv)

The success of neural language models (LMs) on many technological tasks has brought about their potential relevance as scientific theories of language despite some clear differences between LM training and child language acquisition. In this paper we argue that some of the most prominent benchmarks for evaluating the syntactic capacities of LMs may not be sufficiently rigorous. In particular, we show that the template-based benchmarks lack the structural diversity commonly found in the theoretical and psychological studies of language. When trained on small-scale data modeling child language acquisition, the LMs can be readily matched by simple baseline models. We advocate for the use of the readily available, carefully curated datasets that have been evaluated for gradient acceptability by large pools of native speakers and are designed to probe the structural basis of grammar specifically. On one such dataset, the LI-Adger dataset, LMs evaluate sentences in a way inconsistent with human language users. We conclude with suggestions for better connecting LMs with the empirical study of child language acquisition.