# Construction Grammar Provides Unique Insight into Neural Language Models

arXiv: https://arxiv.org/abs/2302.02178 · HTML: https://mlsanity.com/paper/2302.02178/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 4 Feb 2023.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Controllable Sentence Simplification](https://mlsanity.com/paper/1910.02677/) (2019)
- [Frequency Effects on Syntactic Rule Learning in Transformers](https://mlsanity.com/paper/2109.07020/) (2021)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative](https://mlsanity.com/paper/2210.13181/) (2022)
- [CxGBERT: BERT meets Construction Grammar](https://mlsanity.com/paper/2011.04134/) (2020)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings](https://mlsanity.com/paper/1906.03608/) (2019)
- [Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords](https://mlsanity.com/paper/2109.11491/) (2021)
- [Are Representations Built from the Ground Up? An Empirical Examination of Local Composition in Language Models](https://mlsanity.com/paper/2210.03575/) (2022)

## Built on by (most cited first, 13 of 13)

- [Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics](https://mlsanity.com/paper/2311.01386/) (2023)
- [Do Construction Distributions Shape Formal Language Learning In German BabyLMs?](https://mlsanity.com/paper/2503.11593/) (2025)
- [Linguistic Generalizations are not Rules: Impacts on Evaluation of LMs](https://mlsanity.com/paper/2502.13195/) (2025)
- [UCxn: Typologically Informed Annotation of Constructions Atop Universal Dependencies](https://mlsanity.com/paper/2403.17748/) (2024)
- [Constructions are Revealed in Word Distributions](https://mlsanity.com/paper/2503.06048/) (2025)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [Enhancing Language Representation with Constructional Information for Natural Language Understanding](https://mlsanity.com/paper/2306.02819/) (2023)
- [Adverbs, Surprisingly](https://mlsanity.com/paper/2305.19650/) (2023)
- [The Computational Learning of Construction Grammars: State of the Art and Prospective Roadmap](https://mlsanity.com/paper/2407.07606/) (2024)
- [LLMs Learn Constructions That Humans Do Not Know](https://mlsanity.com/paper/2508.16837/) (2025)
- [Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns](https://mlsanity.com/paper/2606.27460/) (2026)
- [Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions](https://mlsanity.com/paper/2605.31586/) (2026)
- [Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality](https://mlsanity.com/paper/2609.34936/) (2026)

## Abstract (arXiv)

Construction Grammar (CxG) has recently been used as the basis for probing studies that have investigated the performance of large pretrained language models (PLMs) with respect to the structure and meaning of constructions. In this position paper, we make suggestions for the continuation and augmentation of this line of research. We look at probing methodology that was not designed with CxG in mind, as well as probing methodology that was designed for specific constructions. We analyse selected previous work in detail, and provide our view of the most important challenges and research questions that this promising new field faces.