mlsanityData to 7 Oct 2026

Frequency Effects on Syntactic Rule Learning in Transformers

arXiv 2109.07020 ↗First posted 14 Sep 2021cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models perform well on a variety of linguistic tasks that require symbolic reasoning, raising the question of whether such models implicitly represent abstract symbols and rules. We investigate this question using the case study of BERT's performance on English subject-verb agreement. Unlike prior work, we train multiple instances of BERT from scratch, allowing us to perform a series of controlled interventions at pre-training time. We show that BERT often generalizes well to subject-verb pairs that never occurred in training, suggesting a degree of rule-governed behavior. We also find, however, that performance is heavily influenced by word frequency, with experiments showing that both the absolute frequency of a verb form, as well as the frequency relative to the alternate inflection, are causally implicated in the predictions BERT makes at inference time. Closer analysis of these frequency effects reveals that BERT's behavior is consistent with a system that correctly applies the SVA rule in general but struggles to overcome strong training priors and to estimate agreement features (singular vs. plural) on infrequent lexical items.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 16 of its 62 references that are in the map. First posted on arXiv 14 Sep 2021.

Builds on

16 of 62 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  5. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  6. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  7. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  8. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  9. 2019Assessing BERT's Syntactic Abilitiescited by 191
  10. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  11. 2020Syntactic Structure from Deep Learningcited by 99
  12. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
4 more
  1. 2019Can neural networks understand monotonicity reasoning?cited by 45
  2. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  3. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  4. 2020Probing Linguistic Systematicitycited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 21; 2023: 9; 2024: 8; 2025: 11; 2026: 8.

59 papers.

Sort:

By area:

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  3. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  4. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  5. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  6. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  7. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  8. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  9. 2022Probing for the Usage of Grammatical Numbercited by 57
  10. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  11. 2022Grammatical Error Correction: A Survey of the State of the Artcited by 43
  12. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
38 more
  1. 2024Filtered Corpus Training (FiCT) Shows that Language Models can Generalize from Indirect Evidencecited by 35
  2. 2023Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stabilitycited by 34
  3. 2023Dissociating language and thought in large language modelscited by 27
  4. 2022Neural reality of argument structure constructionscited by 25
  5. 2023How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biasescited by 23
  6. 2023Impact of Co-occurrence on Factual Knowledge of Large Language Modelscited by 19
  7. 2022Geographic Adaptation of Pretrained Language Modelscited by 14
  8. 2023Construction Grammar Provides Unique Insight into Neural Language Modelscited by 13
  9. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  10. 2024On the Reliability of Large Language Models for Causal Discoverycited by 12
  11. 2023Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semanticscited by 11
  12. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  13. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  14. 2025Child-Directed Language Does Not Consistently Boost Syntax Learning in Language Modelscited by 9
  15. 2021Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledgecited by 8
  16. 2022Crystal Transformer: Self-learning neural language model for Generative and Tinkering Design of Materialscited by 8
  17. 2022longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attackscited by 8
  18. 2023In-Depth Look at Word Filling Societal Bias Measurescited by 8
  19. 2024A systematic framework for generating novel experimental hypotheses from language modelscited by 8
  20. 2022Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreementcited by 7
  21. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  22. 2022A Natural Bias for Language Generation Modelscited by 6
  23. 2025Whither symbols in the era of advanced neural networks?cited by 6
  24. 2024EMTeC: A Corpus of Eye Movements on Machine-Generated Textscited by 5
  25. 2022Probing for targeted syntactic knowledge through grammatical error detectioncited by 4
  26. 2024On the Effect of (Near) Duplicate Subwords in Language Modellingcited by 4
  27. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  28. 2025Rapid Word Learning Through Meta In-Context Learningcited by 4
  29. 2022A computational psycholinguistic evaluation of the syntactic abilities of Galician BERT models at the interface of dependency resolution and training timecited by 3
  30. 2024Investigating grammatical abstraction in language models using few-shot learning of novel noun gendercited by 2
  31. 2025Enhancing LLM Knowledge Learning through Generalizationcited by 2
  32. 2025Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive zijicited by 2
  33. 2022Materials Transformers Language Models for Generative Materials Design: a benchmark studycited by 1
  34. 2023On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translationcited by 1
  35. 2024Can Language Models Induce Grammatical Knowledge from Indirect Evidence?cited by 1
  36. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  37. 2026Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretrainingcited by 1
  38. 2026Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patternscited by 1

50 most cited of 59 All 59 (JSON)