mlsanityData to 14 Oct 2026

Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions

arXiv 2605.31586 ↗First posted 29 May 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Grasping the semantics of rare constructions (form-meaning pairings) has been shown to be a challenging problem that has currently only been solved by the largest LLMs. It remains an open question if open-source models have robust constructional understanding, and if so, what learning dynamics underlie the acquisition of this knowledge. Focusing on a set of rare Paired-Focus constructions in English (e.g. "let alone", "much less"), we construct a novel dataset to test their meanings using both scalar adjectival semantics and general world knowledge. Testing a wide range of models differing in parameter count, architecture, and pretraining dataset size, we find that several modestly sized models are sensitive to both the forms and the meanings of Paired-Focus constructions, though models trained on human-scale data fail at all meaning evaluations. Turning to training dynamics for a set of open-checkpoint models, we find that Paired-Focus understanding emerges later in training than Paired-Focus syntactic knowledge, and that learning of Paired-Focus semantics is correlated with gains in some domains of world knowledge. Overall, our empirical results support the conclusion that modestly sized open-source models can grasp the rare Paired-Focus constructions, and demonstrate a connection between knowledge of Paired-Focus constructions and other meaning domains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 62 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

33 of 62 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  8. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  9. 2024Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokenscited by 153
  10. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 99
  11. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  12. 2024Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNscited by 65
21 more
  1. 2024Elements of World Knowledge (EWoK): A Cognition-Inspired Framework for Evaluating Basic World Knowledge in Language Modelscited by 55
  2. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  3. 2024Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 48
  4. 2020CxGBERT: BERT meets Construction Grammarcited by 32
  5. 2023A Better Way to Do Masked Language Model Scoringcited by 27
  6. 2022Neural reality of argument structure constructionscited by 25
  7. 2023A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Constructioncited by 22
  8. 2019Harnessing the linguistic signal to predict scalar inferencescited by 17
  9. 2020BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representationscited by 15
  10. 2023Construction Grammar Provides Unique Insight into Neural Language Modelscited by 13
  11. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  12. 2024Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasonscited by 13
  13. 2025Do Construction Distributions Shape Formal Language Learning In German BabyLMs?cited by 11
  14. 2025Constructions are Revealed in Word Distributionscited by 6
  15. 2025BabyVLM: Data-Efficient Pretraining of VLMs Inspired by Infant Learningcited by 6
  16. 2024Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMscited by 5
  17. 2025Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructionscited by 5
  18. 2025Construction Identification and Disambiguation Using BERT: A Case Study of NPNcited by 4
  19. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  20. 2024Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmaticscited by 3
  21. 2025Language Models at the Syntax-Semantics Interface: A Case Study of the Long-Distance Binding of Chinese Reflexive zijicited by 2

Built on by

Not cited yet.