# VAQUUM: Are Vague Quantifiers Grounded in Visual Data?

arXiv: https://arxiv.org/abs/2502.11874 · HTML: https://mlsanity.com/paper/2502.11874/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 87 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [TallyQA: Answering Complex Counting Questions](https://mlsanity.com/paper/1810.12440/) (2018)
- [Learning To Count Everything](https://mlsanity.com/paper/2104.08391/) (2021)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences](https://mlsanity.com/paper/2401.10529/) (2024)
- [On the Calibration of Large Language Models and Alignment](https://mlsanity.com/paper/2311.13240/) (2023)
- [What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variability](https://mlsanity.com/paper/2305.11707/) (2023)
- [Learning to Count Anything: Reference-less Class-agnostic Counting with Weak Supervision](https://mlsanity.com/paper/2205.10203/) (2022)
- [Visual Commonsense in Pretrained Unimodal and Multimodal Models](https://mlsanity.com/paper/2205.01850/) (2022)
- [Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks](https://mlsanity.com/paper/2012.12352/) (2020)
- [The Hard Positive Truth about Vision-Language Compositionality](https://mlsanity.com/paper/2409.17958/) (2024)
- [The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models](https://mlsanity.com/paper/2310.15061/) (2023)
- [Naming, Describing, and Quantifying Visual Objects in Humans and LLMs](https://mlsanity.com/paper/2403.06935/) (2024)
- [Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers](https://mlsanity.com/paper/2410.13984/) (2024)
- [Be Precise or Fuzzy: Learning the Meaning of Cardinals and Quantifiers from Vision](https://mlsanity.com/paper/1702.05270/) (2017)

## Built on by (most cited first, 4 of 4)

- [Quantification and object perception in Multimodal Large Language Models and human linguistic cognition](https://mlsanity.com/paper/2511.08126/) (2025)
- [Bears, all bears, and some bears. Language Constraints on Language Models' Inductive Inferences](https://mlsanity.com/paper/2601.09852/) (2026)
- [How Do People Quantify Naturally: Evidence from Mandarin Picture Description](https://mlsanity.com/paper/2602.09838/) (2026)
- [LLMs Infer Cultural Context but Fail to Apply It When Responding](https://mlsanity.com/paper/2606.17688/) (2026)

## Abstract (arXiv)

Vague quantifiers such as "a few" and "many" are influenced by various contextual factors, including the number of objects present in a given context. In this work, we evaluate the extent to which vision-and-language models (VLMs) are compatible with humans when producing or judging the appropriateness of vague quantifiers in visual contexts. We release a novel dataset, VAQUUM, containing 20,300 human ratings on quantified statements across a total of 1089 images. Using this dataset, we compare human judgments and VLM predictions using three different evaluation methods. Our findings show that VLMs, like humans, are influenced by object counts in vague quantifier use. However, we find significant inconsistencies across models in different evaluation settings, suggesting that judging and producing vague quantifiers rely on two different processes.