# CxGBERT: BERT meets Construction Grammar

arXiv: https://arxiv.org/abs/2011.04134 · HTML: https://mlsanity.com/paper/2011.04134/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 30 of 32)

- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://mlsanity.com/paper/2109.14989/) (2021)
- [The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative](https://mlsanity.com/paper/2210.13181/) (2022)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs](https://mlsanity.com/paper/2502.13195/) (2025)
- [Learned Construction Grammars Converge Across Registers Given Increased Exposure](https://mlsanity.com/paper/2110.05663/) (2021)
- [Exposure and Emergence in Usage-Based Grammar: Computational Experiments in 35 Languages](https://mlsanity.com/paper/2211.14160/) (2022)
- [Stability of Syntactic Dialect Classification Over Space and Time](https://mlsanity.com/paper/2209.04958/) (2022)
- [Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena](https://mlsanity.com/paper/2403.06965/) (2024)
- [Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions](https://mlsanity.com/paper/2501.04661/) (2025)
- [Construction Identification and Disambiguation Using BERT: A Case Study of NPN](https://mlsanity.com/paper/2503.18751/) (2025)
- [Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning](https://mlsanity.com/paper/2506.04408/) (2025)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [BERT Has Uncommon Sense: Similarity Ranking for Word Sense BERTology](https://mlsanity.com/paper/2109.09780/) (2021)
- [Enhancing Language Representation with Constructional Information for Natural Language Understanding](https://mlsanity.com/paper/2306.02819/) (2023)
- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)
- [Why transformers are obviously good models of language](https://mlsanity.com/paper/2408.03855/) (2024)
- [Abstraction not Memory: BERT and the English Article System](https://mlsanity.com/paper/2206.04184/) (2022)
- [The Computational Learning of Construction Grammars: State of the Art and Prospective Roadmap](https://mlsanity.com/paper/2407.07606/) (2024)
- [PyFCG: Fluid Construction Grammar in Python](https://mlsanity.com/paper/2505.12920/) (2025)
- [LLMs Learn Constructions That Humans Do Not Know](https://mlsanity.com/paper/2508.16837/) (2025)
- [Evaluating CxG Generalisation in LLMs via Construction-Based NLI Fine Tuning](https://mlsanity.com/paper/2509.16422/) (2025)
- [Leveraging Lora Fine-Tuning and Knowledge Bases for Construction Identification](https://mlsanity.com/paper/2601.13105/) (2026)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [A Method for Learning Large-Scale Computational Construction Grammars from Semantically Annotated Corpora](https://mlsanity.com/paper/2603.12754/) (2026)
- [Can Constructions "SCAN" Compositionality ?](https://mlsanity.com/paper/2509.20074/) (2025)
- [LLMs Know More Than Words: A Genre Study with Syntax, Metaphor & Phonetics](https://mlsanity.com/paper/2512.04957/) (2025)
- ['Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's family](https://mlsanity.com/paper/2604.03673/) (2026)

## Abstract (arXiv)

While lexico-semantic elements no doubt capture a large amount of linguistic information, it has been argued that they do not capture all information contained in text. This assumption is central to constructionist approaches to language which argue that language consists of constructions, learned pairings of a form and a function or meaning that are either frequent or have a meaning that cannot be predicted from its component parts. BERT's training objectives give it access to a tremendous amount of lexico-semantic information, and while BERTology has shown that BERT captures certain important linguistic dimensions, there have been no studies exploring the extent to which BERT might have access to constructional information. In this work we design several probes and conduct extensive experiments to answer this question. Our results allow us to conclude that BERT does indeed have access to a significant amount of information, much of which linguists typically call constructional information. The impact of this observation is potentially far-reaching as it provides insights into what deep learning methods learn from text, while also showing that information contained in constructions is redundantly encoded in lexico-semantics.