mlsanityData to 7 Oct 2026

Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons

arXiv 2403.17760 ↗First posted 26 Mar 2024cs.CLcited by 13

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this paper, we make a contribution that can be understood from two perspectives: from an NLP perspective, we introduce a small challenge dataset for NLI with large lexical overlap, which minimises the possibility of models discerning entailment solely based on token distinctions, and show that GPT-4 and Llama 2 fail it with strong bias. We then create further challenging sub-tasks in an effort to explain this failure. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features. This enables us to probe for LLM's understanding of these constructions in various ways, and we find that they fail in a variety of ways to distinguish between them, suggesting that they don't adequately represent their meaning or capture the lexical properties of phrasal heads.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 4 of its 21 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

4 of 21 references.

References · 4

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spacescited by 13
  4. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blindscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 2.

13 papers.

Sort:

By area:

  1. 2025Linguistic Generalizations are not Rules: Impacts on Evaluation of LMscited by 10
  2. 2025Constructions are Revealed in Word Distributionscited by 6
  3. 2025Beyond Memorization: Assessing Semantic Generalization in Large Language Models Using Phrasal Constructionscited by 5
  4. 2025Construction Identification and Disambiguation Using BERT: A Case Study of NPNcited by 4
  5. 2025Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaningcited by 4
  6. 2024Analysis of Argument Structure Constructions in the Large Language Model BERTcited by 3
  7. 2025Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Inductioncited by 3
  8. 2025Discourse Heuristics For Paradoxically Moral Self-Correctioncited by 3
  9. 2024Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Inductioncited by 2
  10. 2025LLMs Learn Constructions That Humans Do Not Knowcited by 1
  11. 2026Leveraging Lora Fine-Tuning and Knowledge Bases for Construction Identificationcited by 1
  12. 2025BabyLM's First Constructions: Causal probing provides a signal of learningcited by 0
1 more
  1. 2026Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Negation: A Pink Elephant in the Large Language Models' Room?
  2. 2025IMPACT: Inflectional Morphology Probes Across Complex Typologies
  3. 2023Not wacky vs. definitely wacky: A study of scalar adverbs in pretrained language models
  4. 2026Semantic Abstraction for Natural Language Inference: a Methodological Framework for Discovering and Compensating Semantic Knowledge and Reasoning Gaps in Large Language Models
  5. 2023What does the Failure to Reason with "Respectively" in Zero/Few-Shot Settings Tell Us about Language Models?
  6. 2026Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity
  7. 2025Let's CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition
  8. 2025SR-LLM: Rethinking the Structured Representation in Large Language Model