# A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces

arXiv: https://arxiv.org/abs/2305.18598 · HTML: https://mlsanity.com/paper/2305.18598/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 62 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Attention-based Deep Multiple Instance Learning](https://mlsanity.com/paper/1802.04712/) (2018)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [On the proper role of linguistically-oriented deep net analysis in linguistic theorizing](https://mlsanity.com/paper/2106.08694/) (2021)
- [Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT](https://mlsanity.com/paper/2101.11043/) (2021)
- [When classifying grammatical role, BERT doesn't care about word order... except when it matters](https://mlsanity.com/paper/2203.06204/) (2022)
- [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023)
- [(Re)construing Meaning in NLP](https://mlsanity.com/paper/2005.09099/) (2020)
- [Deriving Contextualised Semantic Features from BERT (and Other Transformer Model) Embeddings](https://mlsanity.com/paper/2012.15353/) (2020)
- [ALL Dolphins Are Intelligent and SOME Are Friendly: Probing BERT for Nouns' Semantic Properties and their Prototypicality](https://mlsanity.com/paper/2110.06376/) (2021)
- [Feature2Vec: Distributional semantic modelling of human property knowledge](https://mlsanity.com/paper/1908.11439/) (2019)

## Built on by (most cited first, 13 of 13)

- [Visual Grounding Helps Learn Word Meanings in Low-Data Regimes](https://mlsanity.com/paper/2310.13257/) (2023)
- [Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons](https://mlsanity.com/paper/2403.17760/) (2024)
- [Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling](https://mlsanity.com/paper/2403.14551/) (2024)
- [Constructions are Revealed in Word Distributions](https://mlsanity.com/paper/2503.06048/) (2025)
- [Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructions](https://mlsanity.com/paper/2501.04661/) (2025)
- [Construction Identification and Disambiguation Using BERT: A Case Study of NPN](https://mlsanity.com/paper/2503.18751/) (2025)
- [Analysis of Argument Structure Constructions in the Large Language Model BERT](https://mlsanity.com/paper/2408.04270/) (2024)
- [Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era](https://mlsanity.com/paper/2506.03994/) (2025)
- [Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings](https://mlsanity.com/paper/2508.13729/) (2025)
- [semantic-features: A User-Friendly Tool for Studying Contextual Word Embeddings in Interpretable Semantic Spaces](https://mlsanity.com/paper/2506.06169/) (2025)
- [Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs](https://mlsanity.com/paper/2507.22286/) (2025)
- [LLMs Learn Constructions That Humans Do Not Know](https://mlsanity.com/paper/2508.16837/) (2025)
- [Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions](https://mlsanity.com/paper/2605.31586/) (2026)

## Abstract (arXiv)

We study semantic construal in grammatical constructions using large language models. First, we project contextual word embeddings into three interpretable semantic spaces, each defined by a different set of psycholinguistic feature norms. We validate these interpretable spaces and then use them to automatically derive semantic characterizations of lexical items in two grammatical constructions: nouns in subject or object position within the same sentence, and the AANN construction (e.g., `a beautiful three days'). We show that a word in subject position is interpreted as more agentive than the very same word in object position, and that the nouns in the AANN construction are interpreted as more measurement-like than when in the canonical alternation. Our method can probe the distributional meaning of syntactic constructions at a templatic level, abstracted away from specific lexemes.