# RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency

arXiv: https://arxiv.org/abs/1809.01329 · HTML: https://mlsanity.com/paper/1809.01329/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2018.

## Built on by (most cited first, 23 of 23)

- [Do Large Language Models know what humans know?](https://mlsanity.com/paper/2209.01515/) (2022)
- [Can language models handle recursively nested grammatical structures? A case study on comparing models and humans](https://mlsanity.com/paper/2210.15303/) (2022)
- [A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans](https://mlsanity.com/paper/2106.03232/) (2021)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Accounting for Agreement Phenomena in Sentence Comprehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention](https://mlsanity.com/paper/2104.12874/) (2021)
- [How much pretraining data do language models need to learn syntax?](https://mlsanity.com/paper/2109.03160/) (2021)
- [The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation](https://mlsanity.com/paper/2109.07848/) (2021)
- [SLING: Sino Linguistic Evaluation of Large Language Models](https://mlsanity.com/paper/2210.11689/) (2022)
- [Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models](https://mlsanity.com/paper/2104.07578/) (2021)
- [Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbations](https://mlsanity.com/paper/2109.14017/) (2021)
- [Controlled Evaluation of Syntactic Knowledge in Multilingual Language Models](https://mlsanity.com/paper/2411.07474/) (2024)
- [Can RNNs learn Recursive Nested Subject-Verb Agreements?](https://mlsanity.com/paper/2101.02258/) (2021)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)
- [Characterizing Verbatim Short-Term Memory in Neural Language Models](https://mlsanity.com/paper/2210.13569/) (2022)
- [Learning is Forgetting: LLM Training As Lossy Compression](https://mlsanity.com/paper/2604.07569/) (2026)
- [Clause-Internal or Clause-External? Testing Turkish Reflexive Binding in Adapted versus Chain of Thought Large Language Models](https://mlsanity.com/paper/2602.00380/) (2026)
- [Spelling convention sensitivity in neural language models](https://mlsanity.com/paper/2303.03457/) (2023)
- [Representations as Language: An Information-Theoretic Framework for Interpretability](https://mlsanity.com/paper/2406.02449/) (2024)
- [Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models](https://mlsanity.com/paper/2409.12435/) (2024)
- [Language Models Largely Exhibit Human-like Constituent Ordering Preferences](https://mlsanity.com/paper/2502.05670/) (2025)
- [Assessing the Limits of the Distributional Hypothesis in Semantic Spaces: Trait-based Relational Knowledge and the Impact of Co-occurrences](https://mlsanity.com/paper/2205.07603/) (2022)
- [Interpretable Sentence Representation with Variational Autoencoders and Attention](https://mlsanity.com/paper/2305.02810/) (2023)
- [FAC2E: Better Understanding Large Language Model Capabilities by Dissociating Language and Cognition](https://mlsanity.com/paper/2403.00126/) (2024)

## Abstract (arXiv)

Recurrent neural networks (RNNs) are the state of the art in sequence modeling for natural language. However, it remains poorly understood what grammatical characteristics of natural language they implicitly learn and represent as a consequence of optimizing the language modeling objective. Here we deploy the methods of controlled psycholinguistic experimentation to shed light on to what extent RNN behavior reflects incremental syntactic state and grammatical dependency representations known to characterize human linguistic behavior. We broadly test two publicly available long short-term memory (LSTM) English sequence models, and learn and test a new Japanese LSTM. We demonstrate that these models represent and maintain incremental syntactic state, but that they do not always generalize in the same way as humans. Furthermore, none of our models learn the appropriate grammatical dependency configurations licensing reflexive pronouns or negative polarity items.