mlsanityData to 7 Oct 2026

Assessing BERT's Syntactic Abilities

arXiv 1901.05287 ↗First posted 16 Jan 2019cs.CLcited by 191

Not on the map. Why?

Abstract (arXiv)

I assess the extent to which the recently introduced BERT model captures English syntactic phenomena, using (1) naturally-occurring subject-verb agreement stimuli; (2) "coloreless green ideas" subject-verb agreement stimuli, in which content words in natural sentences are randomly replaced with words sharing the same part-of-speech and inflection; and (3) manually crafted stimuli for subject-verb agreement and reflexive anaphora phenomena. The BERT model performs remarkably well on all cases.

Key facts. Cited by 191 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 66; 2022: 48; 2023: 34; 2024: 21; 2025: 13; 2026: 9.

191 papers.

Sort:

By area:

  1. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  2. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  5. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  6. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  7. 2022A Review on Language Models as Knowledge Basescited by 156
  8. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  9. 2021A Survey on Data Augmentation for Text Classificationcited by 111
  10. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  11. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  12. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
38 more
  1. 2023LLM-Rec: Personalized Recommendation via Prompting Large Language Modelscited by 77
  2. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  3. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  4. 2022Measuring the Mixing of Contextual Information in the Transformercited by 63
  5. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  6. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  7. 2022Probing for the Usage of Grammatical Numbercited by 57
  8. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  9. 2022Pretraining Without Attentioncited by 51
  10. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  11. 2021Differentiable Subset Pruning of Transformer Headscited by 46
  12. 2024Detection and Measurement of Syntactic Templates in Generated Textcited by 44
  13. 2021On the proper role of linguistically-oriented deep net analysis in linguistic theorizingcited by 43
  14. 2023LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Modelcited by 43
  15. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  16. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  17. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
  18. 2021Lexicon Enhanced Chinese Sequence Labeling Using BERT Adaptercited by 34
  19. 2023Do large language models resemble humans in language use?cited by 34
  20. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  21. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  22. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  23. 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
  24. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  25. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  26. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  27. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  28. 2023A Geometric Notion of Causal Probingcited by 23
  29. 2021Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Modelscited by 22
  30. 2021On Explaining Your Explanations of BERT: An Empirical Study with Sequence Classificationcited by 21
  31. 2023Is Bigger and Deeper Always Better? Probing LLaMA Across Scales and Layerscited by 20
  32. 2021Transformers Generalize Linearlycited by 19
  33. 2021Interpreting Deep Learning Models in Natural Language Processing: A Reviewcited by 19
  34. 2021Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpuscited by 18
  35. 2024GPT-3.5 for Grammatical Error Correctioncited by 18
  36. 2023DLAMA: A Framework for Curating Culturally Diverse Facts for Probing the Knowledge of Pretrained Language Modelscited by 17
  37. 2021The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERTcited by 15
  38. 2021Inspecting the concept knowledge graph encoded by modern language modelscited by 15

50 most cited of 191 All 191 (JSON)