mlsanityData to 14 Oct 2026

Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs

arXiv 2502.13195 ↗First posted 18 Feb 2025cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Linguistic evaluations of how well LMs generalize to produce or understand language often implicitly take for granted that natural languages are generated by symbolic rules. According to this perspective, grammaticality is determined by whether sentences obey such rules. Interpretation is compositionally generated by syntactic rules operating on meaningful words. Semantic parsing maps sentences into formal logic. Failures of LMs to obey strict rules are presumed to reveal that LMs do not produce or understand language like humans. Here we suggest that LMs' failures to obey symbolic rules may be a feature rather than a bug, because natural languages are not based on neatly separable, compositional rules. Rather, new utterances are produced and understood by a combination of flexible, interrelated, and context-dependent constructions. Considering gradient factors such as frequencies, context, and function will help us reimagine new benchmarks and analyses to probe whether and how LMs capture the rich, flexible generalizations that comprise natural languages.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 33 of its 114 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

33 of 114 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2018Neural Network Acceptability Judgmentscited by 958
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  10. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  11. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  12. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
21 more
  1. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  2. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 55
  3. 2024Language models align with human judgments on key grammatical constructionscited by 49
  4. 2023Testing AI on language comprehension tasks reveals insensitivity to underlying meaningcited by 45
  5. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
  6. 2020Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigationcited by 34
  7. 2020CxGBERT: BERT meets Construction Grammarcited by 32
  8. 2022Neural reality of argument structure constructionscited by 25
  9. 2021NOPE: A Corpus of Naturally-Occurring Presuppositions in Englishcited by 23
  10. 2020Investigating representations of verb bias in neural language modelscited by 22
  11. 2023SLOG: A Structural Generalization Benchmark for Semantic Parsingcited by 22
  12. 2017Universal Semantic Parsingcited by 21
  13. 2022Naturalistic Causal Probing for Morpho-Syntaxcited by 19
  14. 2023Construction Grammar Provides Unique Insight into Neural Language Modelscited by 13
  15. 2024Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasonscited by 13
  16. 2023From chocolate bunny to chocolate crocodile: Do Language Models Understand Noun Compounds?cited by 12
  17. 2024Manipulating language models' training data to study syntactic constraint learning: the case of English passivizationcited by 12
  18. 2024Controlled Evaluation of Syntactic Knowledge in Multilingual Language Modelscited by 10
  19. 2019Analyzing machine-learned representations: A natural language case studycited by 5
  20. 2024Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformationscited by 4
  21. 2024Is artificial intelligence still intelligence? LLMs generalize to novel adjective-noun pairs, but don't mimic the full human distributioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  2. 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
  3. 2025Model Misalignment and Language Change: Traces of AI-Associated Language in Unscripted Spoken Englishcited by 4
  4. 2025Discourse Heuristics For Paradoxically Moral Self-Correctioncited by 3
  5. 2026Morphemes Without Borders: Evaluating Root-Pattern Morphology in Arabic Tokenizers and LLMscited by 1
  6. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
  7. 2026CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Modelscited by 0
  8. 2026Child-directed speech facilitates production, not comprehension, in BabyLMscited by 0
  9. 2026Similarity All The Way Up: Multilingual Generalization in LLMs Relies on Language-Level Similarity Structurescited by 0
  10. 2026Which Forms of Caregiver Feedback Support Grammar Learning? A Reinforcement-Learning Study of Child-Like Language Modelscited by 0