mlsanityData to 7 Oct 2026

A Systematic Assessment of Syntactic Generalization in Neural Language Models

arXiv 2005.03692 ↗First posted 7 May 2020cs.CLcited by 163

Not on the map. Why?

Abstract (arXiv)

While state-of-the-art neural network models continue to achieve lower perplexity scores on language modeling benchmarks, it remains unknown whether optimizing for broad-coverage predictive performance leads to human-like syntactic knowledge. Furthermore, existing work has not provided a clear picture about the model properties required to produce proper syntactic generalizations. We present a systematic evaluation of the syntactic knowledge of neural language models, testing 20 combinations of model types and data sizes on a set of 34 English-language syntactic test suites. We find substantial differences in syntactic generalization performance by model architecture, with sequential models underperforming other architectures. Factorially manipulating model architecture and training dataset size (1M--40M words), we find that variability in syntactic generalization performance is substantially greater by architecture than by dataset size for the corpora tested in our experiments. Our results also reveal a dissociation between perplexity and syntactic generalization performance.

Key facts. Cited by 163 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 28; 2023: 24; 2024: 33; 2025: 31; 2026: 22.

163 papers.

Sort:

By area:

  1. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  2. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  3. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  4. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  5. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  6. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  7. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  8. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  9. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  10. 2024Language models align with human judgments on key grammatical constructionscited by 49
  11. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  12. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 44
38 more
  1. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  2. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  3. 2023Prompting is not a substitute for probability measurements in large language modelscited by 43
  4. 2023CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Modelscited by 43
  5. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  6. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  7. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  8. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  9. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  10. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  11. 2024Mission: Impossible Language Modelscited by 31
  12. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  13. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  14. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  15. 2023Dissociating language and thought in large language modelscited by 27
  16. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 27
  17. 2021Structural Guidance for Transformer Language Modelscited by 25
  18. 2022Neural reality of argument structure constructionscited by 25
  19. 2021How is BERT surprised? Layerwise detection of linguistic anomaliescited by 24
  20. 2022Probing for Incremental Parse States in Autoregressive Language Modelscited by 24
  21. 2024Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Modelscited by 24
  22. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  23. 2023Inductive reasoning in humans and large language modelscited by 23
  24. 2025Can Language Models Learn Typologically Implausible Languages?cited by 23
  25. 2022When a sentence does not introduce a discourse entity, Transformer-based models still sometimes refer to itcited by 22
  26. 2022Language model acceptability judgements are not always robust to contextcited by 22
  27. 2023How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speechcited by 22
  28. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  29. 2021How much pretraining data do language models need to learn syntax?cited by 21
  30. 2025Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructionscited by 21
  31. 2021Transformers Generalize Linearlycited by 19
  32. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
  33. 2021Learning Hierarchical Structures with Differentiable Nondeterministic Stackscited by 16
  34. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  35. 2025What Can String Probability Tell Us About Grammaticality?cited by 16
  36. 2021Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Modelscited by 15
  37. 2023Evaluating Neural Language Models as Cognitive Models of Language Acquisitioncited by 15
  38. 2024Investigating Critical Period Effects in Language Acquisition through Neural Language Modelscited by 15

50 most cited of 163 All 163 (JSON)