mlsanityData to 7 Oct 2026

Construction Identification and Disambiguation Using BERT: A Case Study of NPN

arXiv 2503.18751 ↗First posted 24 Mar 2025cs.CL cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Construction Grammar hypothesizes that knowledge of a language consists chiefly of knowledge of form-meaning pairs (''constructions'') that include vocabulary, general grammar rules, and even idiosyncratic patterns. Recent work has shown that transformer language models represent at least some constructional patterns, including ones where the construction is rare overall. In this work, we probe BERT's representation of the form and meaning of a minor construction of English, the NPN (noun-preposition-noun) construction -- exhibited in such expressions as face to face and day to day -- which is known to be polysemous. We construct a benchmark dataset of semantically annotated corpus instances (including distractors that superficially resemble the construction). With this dataset, we train and evaluate probing classifiers. They achieve decent discrimination of the construction from distractors, as well as sense disambiguation among true instances of the construction, revealing that BERT embeddings carry indications of the construction's semantics. Moreover, artificially permuting the word order of true construction instances causes them to be rejected, indicating sensitivity to matters of form. We conclude that BERT does latently encode at least some knowledge of the NPN construction going beyond a surface syntactic pattern and lexical cues.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 28 of its 60 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

28 of 60 references.

References · 28

  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  7. 2019Designing and Interpreting Probes with Control Taskscited by 608
  8. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  9. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  10. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  11. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  12. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
16 more
  1. 2019Assessing BERT's Syntactic Abilitiescited by 191
  2. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  3. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  4. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 119
  5. 2020Syntactic Structure from Deep Learningcited by 99
  6. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  7. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  8. 2020CxGBERT: BERT meets Construction Grammarcited by 32
  9. 2022Neural reality of argument structure constructionscited by 25
  10. 2023A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Constructioncited by 22
  11. 2018Comprehensive Supersense Disambiguation of English Prepositions and Possessivescited by 20
  12. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  13. 2024Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasonscited by 13
  14. 2021Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowordscited by 9
  15. 2024UCxn: Typologically Informed Annotation of Constructions Atop Universal Dependenciescited by 7
  16. 2023CALaMo: a Constructionist Assessment of Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  2. 2026Do Language Models Know What Not to Say? Causal Evidence for Statistical Preemption in LLMscited by 1
  3. 2026'Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's familycited by 0
  4. 2026Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructionscited by 0