mlsanityData to 7 Oct 2026

Targeted Syntactic Evaluation of Language Models

arXiv 1808.09031 ↗First posted 27 Aug 2018cs.CLcited by 239

Not on the map. Why?

Abstract (arXiv)

We present a dataset for evaluating the grammaticality of the predictions of a language model. We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence. The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items. We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one. In an experiment using this data set, an LSTM language model performed poorly on many of the constructions. Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM's accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.

Key facts. Cited by 239 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 47; 2023: 35; 2024: 38; 2025: 35; 2026: 33.

239 papers.

Sort:

By area:

  1. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  2. 2023Explainability for Large Language Models: A Surveycited by 331
  3. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  4. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  5. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  6. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  7. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  8. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 86
  9. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  10. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  11. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
  12. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
38 more
  1. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  2. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
  3. 2022Probing for the Usage of Grammatical Numbercited by 57
  4. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  5. 2022Pretraining Without Attentioncited by 51
  6. 2024Language models align with human judgments on key grammatical constructionscited by 49
  7. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  8. 2021CLiMP: A Benchmark for Chinese Language Model Evaluationcited by 44
  9. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  10. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  11. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  12. 2021Word Acquisition in Neural Language Modelscited by 41
  13. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  14. 2022Characterizing Intrinsic Compositionality in Transformers with Tree Projectionscited by 40
  15. 2023Modeling rapid language learning by distilling Bayesian priors into artificial neural networkscited by 40
  16. 2023Neither hype nor gloom do DNNs justicecited by 40
  17. 2021The Low-Dimensional Linear Geometry of Contextualized Word Representationscited by 39
  18. 2021Evaluating Saliency Methods for Neural Language Modelscited by 37
  19. 2022Analyzing Encoded Concepts in Transformer Language Modelscited by 37
  20. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  21. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 36
  22. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
  23. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  24. 2023Do large language models resemble humans in language use?cited by 34
  25. 2021A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humanscited by 32
  26. 2024Mission: Impossible Language Modelscited by 31
  27. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  28. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  29. 2021Sensitivity as a Complexity Measure for Sequence Classification Taskscited by 28
  30. 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
  31. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  32. 2025Language Models Fail to Introspect About Their Knowledge of Languagecited by 28
  33. 2023Dissociating language and thought in large language modelscited by 27
  34. 2022RuCoLA: Russian Corpus of Linguistic Acceptabilitycited by 26
  35. 2021DirectProbe: Studying Representations without Classifierscited by 25
  36. 2021Structural Guidance for Transformer Language Modelscited by 25
  37. 2022Probing for Incremental Parse States in Autoregressive Language Modelscited by 24
  38. 2024Decoding Probing: Revealing Internal Linguistic Structures in Neural Language Models using Minimal Pairscited by 24

50 most cited of 239 All 239 (JSON)