# The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative

arXiv: https://arxiv.org/abs/2210.13181 · HTML: https://mlsanity.com/paper/2210.13181/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Assessing BERT's Syntactic Abilities](https://mlsanity.com/paper/1901.05287/) (2019)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Probing Pretrained Language Models for Lexical Semantics](https://mlsanity.com/paper/2010.05731/) (2020)
- [Frequency Effects on Syntactic Rule Learning in Transformers](https://mlsanity.com/paper/2109.07020/) (2021)
- [CxGBERT: BERT meets Construction Grammar](https://mlsanity.com/paper/2011.04134/) (2020)
- [Learning to Recognize Dialect Features](https://mlsanity.com/paper/2010.12707/) (2020)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [Frequency vs. Association for Constraint Selection in Usage-Based Construction Grammar](https://mlsanity.com/paper/1904.05529/) (2019)

## Built on by (most cited first, 30 of 35)

- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [Large Language Models in Education: Vision and Opportunities](https://mlsanity.com/paper/2311.13160/) (2023)
- [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs](https://mlsanity.com/paper/2502.13195/) (2025)
- [UCxn: Typologically Informed Annotation of Constructions Atop Universal Dependencies](https://mlsanity.com/paper/2403.17748/) (2024)
- [Constructions are Revealed in Word Distributions](https://mlsanity.com/paper/2503.06048/) (2025)
- [Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena](https://mlsanity.com/paper/2403.06965/) (2024)
- [Probing the Category of Verbal Aspect in Transformer Language Models](https://mlsanity.com/paper/2406.02335/) (2024)
- [Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set](https://mlsanity.com/paper/2503.10515/) (2025)
- [Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity](https://mlsanity.com/paper/2305.18185/) (2023)
- [Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently](https://mlsanity.com/paper/2401.06640/) (2024)
- [Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics](https://mlsanity.com/paper/2404.03301/) (2024)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [Enhancing Language Representation with Constructional Information for Natural Language Understanding](https://mlsanity.com/paper/2306.02819/) (2023)
- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)
- [Unleashing the Multilingual Encoder Potential: Boosting Zero-Shot Performance via Probability Calibration](https://mlsanity.com/paper/2310.05069/) (2023)
- [What do Transformers Know about Government?](https://mlsanity.com/paper/2404.14270/) (2024)
- [The Computational Learning of Construction Grammars: State of the Art and Prospective Roadmap](https://mlsanity.com/paper/2407.07606/) (2024)
- [PyFCG: Fluid Construction Grammar in Python](https://mlsanity.com/paper/2505.12920/) (2025)
- [From Monolingual to Bilingual: Investigating Language Conditioning in Large Language Models for Psycholinguistic Tasks](https://mlsanity.com/paper/2508.02502/) (2025)
- [LLMs Learn Constructions That Humans Do Not Know](https://mlsanity.com/paper/2508.16837/) (2025)
- [Leveraging Lora Fine-Tuning and Knowledge Bases for Construction Identification](https://mlsanity.com/paper/2601.13105/) (2026)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [A Method for Learning Large-Scale Computational Construction Grammars from Semantically Annotated Corpora](https://mlsanity.com/paper/2603.12754/) (2026)
- [Convergent Representations of Linguistic Constructions in Human and Artificial Neural Systems](https://mlsanity.com/paper/2603.29617/) (2026)
- [Do Large Language Models know who did what to whom?](https://mlsanity.com/paper/2504.16884/) (2025)
- [BabyLM's First Constructions: Causal probing provides a signal of learning](https://mlsanity.com/paper/2506.02147/) (2025)
- [CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models](https://mlsanity.com/paper/2602.21978/) (2026)

## Abstract (arXiv)

Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of language, i.e., linguistic units of different granularity that combine syntax and semantics. As a first step towards assessing the compatibility of CxG with the syntactic and semantic knowledge demonstrated by state-of-the-art pretrained language models (PLMs), we present an investigation of their capability to classify and understand one of the most commonly studied constructions, the English comparative correlative (CC). We conduct experiments examining the classification accuracy of a syntactic probe on the one hand and the models' behaviour in a semantic application task on the other, with BERT, RoBERTa, and DeBERTa as the example PLMs. Our results show that all three investigated PLMs are able to recognise the structure of the CC but fail to use its meaning. While human-like performance of PLMs on many NLP tasks has been alleged, this indicates that PLMs still suffer from substantial shortcomings in central domains of linguistic knowledge.