# Naming, Describing, and Quantifying Visual Objects in Humans and LLMs

arXiv: https://arxiv.org/abs/2403.06935 · HTML: https://mlsanity.com/paper/2403.06935/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 37 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 8 of 8)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Grounding Language Models to Images for Multimodal Inputs and Outputs](https://mlsanity.com/paper/2301.13823/) (2023)

## Built on by (most cited first, 3 of 3)

- [VAQUUM: Are Vague Quantifiers Grounded in Visual Data?](https://mlsanity.com/paper/2502.11874/) (2025)
- [SceneGram: Conceptualizing and Describing Tangrams in Scene Context](https://mlsanity.com/paper/2506.11631/) (2025)
- [Quantification and object perception in Multimodal Large Language Models and human linguistic cognition](https://mlsanity.com/paper/2511.08126/) (2025)

## Abstract (arXiv)

While human speakers use a variety of different expressions when describing the same object in an image, giving rise to a distribution of plausible labels driven by pragmatic constraints, the extent to which current Vision & Language Large Language Models (VLLMs) can mimic this crucial feature of language use is an open question. This applies to common, everyday objects, but it is particularly interesting for uncommon or novel objects for which a category label may be lacking or fuzzy. Furthermore, similar patterns of variation are observed among human speakers for highly context-sensitive expressions, such as the quantifiers 'few' or 'most'. In our work, we evaluate VLLMs (FROMAGe, BLIP-2, LLaVA) on three categories (nouns, attributes, and quantifiers) where humans show great subjective variability concerning the distribution over plausible labels, using datasets and resources mostly under-explored in previous work. Our results reveal mixed evidence on the ability of VLLMs to capture human naming preferences at generation time: while some models are good at mimicking human distributions for nouns and attributes, all of them fail to assign quantifiers, a task that requires more accurate, high-level reasoning.