Using Perspectival Words Is Harder Than Vocabulary Words for Humans and Even More So for Multimodal Language Models
Not on the map. Why?
Abstract (arXiv)
Multimodal language models (MLMs) increasingly demonstrate human-like communication, yet their use of everyday perspectival words remains poorly understood. To address this gap, we compare humans and MLMs in their use of three word types that impose increasing cognitive demands: vocabulary (for example, "boat" or "cup"), possessives (for example, "mine" versus "yours"), and demonstratives (for example, "this one" versus "that one"). Testing seven MLMs against human participants, we find that perspectival words are harder than vocabulary words for both groups. The gap is larger for MLMs: while models approach human-level performance on vocabulary, they show clear deficits with possessives and even greater difficulty with demonstratives. Ablation analyses indicate that limitations in perspective-taking and spatial reasoning are key sources of these gaps. Instruction-based prompting reduces the gap for possessives but leaves demonstratives far below human performance. These results show that, unlike vocabulary, perspectival words pose a greater challenge in human communication, and this difficulty is amplified in MLMs, revealing a shortfall in their pragmatic and social-cognitive abilities.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 57 references that are in the map. First posted on arXiv 29 May 2025.
Builds on
17 of 57 references.
References · 17
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
- 2016Visual Dialogcited by 377
- 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
- 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
- 2020Experience Grounds Languagecited by 242
- 2023Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Modelscited by 163
- 2016GuessWhat?! Visual object discovery through multi-modal dialoguecited by 113
- 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
- 2020Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESuppositioncited by 77
5 more
- 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 70
- 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
- 2024PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilitiescited by 59
- 2024RAcQUEt: Unveiling the Dangers of Overlooked Referential Ambiguity in Visual LLMscited by 8
- 2018Ask No More: Deciding when to guess in referential visual dialoguecited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Pseudowords as probes: Large Language Models show little of the sublexical sensitivity that governs human pseudoword processing
- 2025Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans
- 2025The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect
- 2025Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings
- 2025Evidence of conceptual mastery in the application of rules by Large Language Models
- 2024From Babbling to Fluency: Evaluating the Evolution of Language Models in Terms of Human Language Acquisition
- 2025Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback
- 2025Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings