mlsanityData to 14 Oct 2026

Constructions are Revealed in Word Distributions

arXiv 2503.06048 ↗First posted 8 Mar 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Construction grammar posits that constructions, or form-meaning pairings, are acquired through experience with language (the distributional learning hypothesis). But how much information about constructions does this distribution actually contain? Corpus-based analyses provide some answers, but text alone cannot answer counterfactual questions about what caused a particular word to occur. This requires computable models of the distribution over strings -- namely, pretrained language models (PLMs). Here, we treat a RoBERTa model as a proxy for this distribution and hypothesize that constructions will be revealed within it as patterns of statistical affinity. We support this hypothesis experimentally: many constructions are robustly distinguished, including (i) hard cases where semantically distinct constructions are superficially similar, as well as (ii) schematic constructions, whose ``slots'' can be filled by abstract word classes. Despite this success, we also provide qualitative evidence that statistical affinity alone may be insufficient to identify all constructions from text. Thus, statistical affinity is likely an important, but partial, signal available to learners.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 28 of its 89 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

28 of 89 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  7. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  8. 2019Designing and Interpreting Probes with Control Taskscited by 640
  9. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  10. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  11. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  12. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
16 more
  1. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  2. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
  3. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  4. 2022The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlativecited by 35
  5. 2022Neural reality of argument structure constructionscited by 25
  6. 2024Language Models as Models of Languagecited by 24
  7. 2020Investigating representations of verb bias in neural language modelscited by 22
  8. 2023A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Constructioncited by 22
  9. 2021Idiomatic Expression Identification using Semantic Compatibilitycited by 19
  10. 2023Construction Grammar Provides Unique Insight into Neural Language Modelscited by 13
  11. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  12. 2024Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasonscited by 13
  13. 2019Frequency vs. Association for Constraint Selection in Usage-Based Construction Grammarcited by 11
  14. 2021Linguistic Dependencies and Statistical Dependencecited by 10
  15. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blindscited by 7
  16. 2024Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomenacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
  2. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  3. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  4. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
  5. 2026Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructionscited by 0
  6. 2026Child-directed speech facilitates production, not comprehension, in BabyLMscited by 0