mlsanityData to 14 Oct 2026

Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Task

arXiv 2605.31480 ↗First posted 29 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Do neural models, such as Large Language Models, genuinely acquire compositional abilities for interpretation of natural language? When we talk about semantic interpretation, we can distinguish two complementary aspects: establishing what an expression refers to in the world (which we call the Extensional task) and representing its sense in a structured way (which we call the Intensional task). We evaluate LLMs and humans on both tasks in the setting of the Personal Relation Task (Paperno 2022) in which, given a universe of people and their relationships with each other, one is asked to interpret a noun phrase such as "Amber's parent's friend". Here, for the Intensional task, the answer is the formula "friend(parent(amber))", and for the Extensional task, the person. We find that humans and LLMs show opposite strengths: humans perform better on Extensional than Intensional tasks, and LLMs vice versa. Our methodology brings greater nuance to the understanding of compositional abilities in modern machine learning models. Our results support the notion that the lack of referential grounding in LLM training is a crucial missing component in mimicking human-like language understanding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 48 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

20 of 48 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  6. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  7. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 284
  8. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 263
  9. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  10. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  11. 2019Distributional Semantics and Linguistic Theorycited by 53
  12. 2019Human few-shot learning of compositional instructionscited by 47
8 more
  1. 2022Structural generalization is hard for sequence-to-sequence modelscited by 24
  2. 2024A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practicescited by 15
  3. 2023What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structurecited by 13
  4. 2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
  5. 2024Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representationscited by 6
  6. 2024PMB5: Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarkscited by 5
  7. 2018On learning an interpreted language with recurrent modelscited by 3
  8. 2024From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisitioncited by 2

Built on by

Not cited yet.