mlsanityData to 14 Oct 2026

The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

arXiv 2210.13181 ↗First posted 24 Oct 2022cs.CLcited by 35

Not on the map. Why?

Abstract (arXiv)

Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of different granularity that combine syntax and semantics. As a first step towards assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of-the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC). We conduct experiments examining the classification accuracy of a syntactic probe on the one hand and the models' behaviour in a semantic application task on the other, with BERT, RoBERTa, and DeBERTa as the example PLMs. Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning. While human-like performance of PLMs on many NLP tasks has been alleged, this indicates that PLMs still suffer from substantial shortcomings in central domains of linguistic knowledge.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Builds on 24 of its 52 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

24 of 52 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018Deep contextualized word representationscited by 2,290
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  11. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
12 more
  1. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  2. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  3. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  4. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  5. 2019Assessing BERT's Syntactic Abilitiescited by 192
  6. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  7. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  8. 2021Frequency Effects on Syntactic Rule Learning in Transformerscited by 59
  9. 2020CxGBERT: BERT meets Construction Grammarcited by 32
  10. 2020Learning to Recognize Dialect Featurescited by 29
  11. 2022Neural reality of argument structure constructionscited by 25
  12. 2019Frequency vs. Association for Constraint Selection in Usage-Based Construction Grammarcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 9; 2025: 8; 2026: 10.

35 papers.

Sort:

By area:

  1. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  2. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  3. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  4. 2023A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Constructioncited by 22
  5. 2023Construction Grammar Provides Unique Insight into Neural Language Modelscited by 13
  6. 2025Linguistic Generalizations are not Rules: Impacts on Evaluation of LMscited by 10
  7. 2024UCxn: Typologically Informed Annotation of Constructions Atop Universal Dependenciescited by 7
  8. 2025Constructions are Revealed in Word Distributionscited by 6
  9. 2024Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomenacited by 5
  10. 2024Probing the Category of Verbal Aspect in Transformer Language Modelscited by 5
  11. 2025Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Setcited by 4
  12. 2023Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivitycited by 3
23 more
  1. 2024Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistentlycited by 3
  2. 2024Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmaticscited by 3
  3. 2024Analysis of Argument Structure Constructions in the Large Language Model BERTcited by 3
  4. 2023Enhancing Language Representation with Constructional Information for Natural Language Understandingcited by 2
  5. 2023Construction Grammar and Language Modelscited by 2
  6. 2023Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibrationcited by 2
  7. 2024What do Transformers Know about Government?cited by 1
  8. 2024The Computational Learning of Construction Grammars: State of the Art and Prospective Roadmapcited by 1
  9. 2025PyFCG: Fluid Construction Grammar in Pythoncited by 1
  10. 2025From Monolingual to Bilingual: Investigating Language Conditioning in Large Language Models for Psycholinguistic Taskscited by 1
  11. 2025LLMs Learn Constructions That Humans Do Not Knowcited by 1
  12. 2026Leveraging Lora Fine-Tuning and Knowledge Bases for Construction Identificationcited by 1
  13. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  14. 2026A Method for Learning Large-Scale Computational Construction Grammars from Semantically Annotated Corporacited by 1
  15. 2026Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systemscited by 1
  16. 2025Do Large Language Models know who did what to whom?cited by 0
  17. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
  18. 2026CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Modelscited by 0
  19. 2026Exploiting the English Grammar Profile for L2 grammatical analysis with LLMscited by 0
  20. 2026'Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's familycited by 0
  21. 2026Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairscited by 0
  22. 2026Linguistic Productivity in Large Language Models: Models Coerce, but do not Preemptcited by 0
  23. 2026Syntax vs. Semantics: How Transformers Learn Deep Dependenciescited by 0