mlsanityData to 7 Oct 2026

Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State

arXiv 1903.03260 ↗First posted 8 Mar 2019cs.CLcited by 137

Not on the map. Why?

Abstract (arXiv)

We deploy the methods of controlled psycholinguistic experimentation to shed light on the extent to which the behavior of neural network language models reflects incremental representations of syntactic state. To do so, we examine model behavior on artificial sentences containing a variety of syntactically complex structures. We test four models: two publicly available LSTM sequence models of English (Jozefowicz et al., 2016; Gulordava et al., 2018) trained on large datasets; an RNNG (Dyer et al., 2016) trained on a small, parsed dataset; and an LSTM trained on the same small corpus as the RNNG. We find evidence that the LSTMs trained on large datasets represent syntactic state over large spans of text in a way that is comparable to the RNNG, while the LSTM trained on the small dataset does not or does so only weakly.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 15; 2023: 22; 2024: 28; 2025: 22; 2026: 23.

137 papers.

Sort:

By area:

  1. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  2. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  3. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  4. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  5. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 50
  6. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  7. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  8. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  9. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  10. 2021Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERTcited by 41
  11. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  12. 2021Word Acquisition in Neural Language Modelscited by 41
38 more
  1. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  2. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  3. 2024Can LLMs Replace Manual Annotation of Software Engineering Artifacts?cited by 40
  4. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  5. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  6. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 34
  7. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  8. 2023Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameterscited by 32
  9. 2024Mission: Impossible Language Modelscited by 31
  10. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  11. 2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
  12. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  13. 2021Sorting through the noise: Testing robustness of information processing in pre-trained language modelscited by 27
  14. 2021Structural Guidance for Transformer Language Modelscited by 25
  15. 2022Probing for Incremental Parse States in Autoregressive Language Modelscited by 24
  16. 2024Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Modelscited by 24
  17. 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 22
  18. 2023A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Constructioncited by 22
  19. 2023Information Value: Measuring Utterance Predictability as Distance from Plausible Alternativescited by 22
  20. 2021How much pretraining data do language models need to learn syntax?cited by 21
  21. 2024Do Language Models Exhibit Human-like Structural Priming Effects?cited by 21
  22. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
  23. 2021So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgementscited by 15
  24. 2021The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generationcited by 15
  25. 2022COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Modelscited by 15
  26. 2024Language in Vivo vs. in Silico: Size Matters but Larger Language Models Still Do Not Comprehend Language on a Par with Humans Due to Impenetrable Semantic Referencecited by 15
  27. 2021Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Modelscited by 14
  28. 2021Model Explainability in Deep Learning Based Natural Language Processingcited by 14
  29. 2024A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Rolescited by 14
  30. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  31. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  32. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 13
  33. 2023MELA: Multilingual Evaluation of Linguistic Acceptabilitycited by 13
  34. 2024Predictions from language models for multiple-choice tasks are not robust under variation of scoring methodscited by 13
  35. 2025LLMs as a synthesis between symbolic and distributed approaches to languagecited by 13
  36. 2024On the Proper Treatment of Tokenization in Psycholinguisticscited by 12
  37. 2024Generalizations across filler-gap dependencies in neural language modelscited by 12
  38. 2023JCoLA: Japanese Corpus of Linguistic Acceptabilitycited by 11

50 most cited of 137 All 137 (JSON)