# LLMs Learn Constructions That Humans Do Not Know

arXiv: https://arxiv.org/abs/2508.16837 · HTML: https://mlsanity.com/paper/2508.16837/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 27 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Builds on (most cited first, 17 of 17)

- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?](https://mlsanity.com/paper/2212.12131/) (2022)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times](https://mlsanity.com/paper/2402.02255/) (2024)
- [The Better Your Syntax, the Better Your Semantics? Probing Pretrained Language Models for the English Comparative Correlative](https://mlsanity.com/paper/2210.13181/) (2022)
- [CxGBERT: BERT meets Construction Grammar](https://mlsanity.com/paper/2011.04134/) (2020)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [When classifying grammatical role, BERT doesn't care about word order... except when it matters](https://mlsanity.com/paper/2203.06204/) (2022)
- [Investigating representations of verb bias in neural language models](https://mlsanity.com/paper/2010.02375/) (2020)
- [A Discerning Several Thousand Judgments: GPT-3 Rates the Article + Adjective + Numeral + Noun Construction](https://mlsanity.com/paper/2301.12564/) (2023)
- [WinoDict: Probing language models for in-context word acquisition](https://mlsanity.com/paper/2209.12153/) (2022)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces](https://mlsanity.com/paper/2305.18598/) (2023)
- [Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons](https://mlsanity.com/paper/2403.17760/) (2024)
- [Finding Variants for Construction-Based Dialectometry: A Corpus-Based Approach to Regional CxGs](https://mlsanity.com/paper/2104.01299/) (2021)
- [Production vs Perception: The Role of Individuality in Usage-Based Grammar Induction](https://mlsanity.com/paper/2104.09033/) (2021)
- [Large corpora and large language models: a replicable method for automating grammatical annotation](https://mlsanity.com/paper/2411.11260/) (2024)

## Built on by (most cited first, 1 of 1)

- ['Layer su Layer': Identifying and Disambiguating the Italian NPN Construction in BERT's family](https://mlsanity.com/paper/2604.03673/) (2026)

## Abstract (arXiv)

This paper investigates false positive constructions: grammatical structures which an LLM hallucinates as distinct constructions but which human introspection does not support. Both a behavioural probing task using contextual embeddings and a meta-linguistic probing task using prompts are included, allowing us to distinguish between implicit and explicit linguistic knowledge. Both methods reveal that models do indeed hallucinate constructions. We then simulate hypothesis testing to determine what would have happened if a linguist had falsely hypothesized that these hallucinated constructions do exist. The high accuracy obtained shows that such false hypotheses would have been overwhelmingly confirmed. This suggests that construction probing methods suffer from a confirmation bias and raises the issue of what unknown and incorrect syntactic knowledge these models also possess.