# The quasi-semantic competence of LLMs: a case study on the part-whole relation

arXiv: https://arxiv.org/abs/2504.02395 · HTML: https://mlsanity.com/paper/2504.02395/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations](https://mlsanity.com/paper/2408.10920/) (2024)
- [Event knowledge in large language models: the gap between the impossible and the unlikely](https://mlsanity.com/paper/2212.01488/) (2022)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [Log Probabilities Are a Reliable Estimate of Semantic Plausibility in Base and Instruction-Tuned Language Models](https://mlsanity.com/paper/2403.14859/) (2024)
- [Do language models have coherent mental models of everyday things?](https://mlsanity.com/paper/2212.10029/) (2022)
- [Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models](https://mlsanity.com/paper/2305.17311/) (2023)
- [BEAR: A Unified Framework for Evaluating Relational Knowledge in Causal and Masked Language Models](https://mlsanity.com/paper/2404.04113/) (2024)
- [An Investigation of LLMs' Inefficacy in Understanding Converse Relations](https://mlsanity.com/paper/2310.05163/) (2023)
- [Understanding Natural Language Understanding Systems. A Critical Analysis](https://mlsanity.com/paper/2303.04229/) (2023)

## Built on by (most cited first, 1 of 1)

- [Relation Geometry in Semantic Space of Language Models](https://mlsanity.com/paper/2607.26762/) (2026)

## Abstract (arXiv)

Understanding the extent and depth of the semantic competence of Large Language Models (LLMs) is at the center of the current scientific agenda in Artificial Intelligence (AI) and Computational Linguistics (CL). We contribute to this endeavor by investigating their knowledge of the part-whole relation, a.k.a. meronymy, which plays a crucial role in lexical organization, but it is significantly understudied. We used data from ConceptNet relations citepspeer2016conceptnet and human-generated semantic feature norms citepMcRae:2005 to explore the abilities of LLMs to deal with textitpart-whole relations. We employed several methods based on three levels of analysis: i.) behavioral testing via prompting, where we directly queried the models on their knowledge of meronymy, ii.) sentence probability scoring, where we tested models' abilities to discriminate correct (real) and incorrect (asymmetric counterfactual) textitpart-whole relations, and iii.) concept representation analysis in vector space, where we proved the linear organization of the textitpart-whole concept in the embedding and unembedding spaces. These analyses present a complex picture that reveals that the LLMs' knowledge of this relation is only partial. They have just a ``quasi-semantic'' competence and still fall short of capturing deep inferential properties.