# HoloCount: A Holistic Visual Counting Benchmark for MLLMs

arXiv: https://arxiv.org/abs/2607.06420 · HTML: https://mlsanity.com/paper/2607.06420/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 48 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds](https://mlsanity.com/paper/1808.01050/) (2018)
- [Drone-based Object Counting by Spatially Regularized Regional Proposal Network](https://mlsanity.com/paper/1707.05972/) (2017)
- [TallyQA: Answering Complex Counting Questions](https://mlsanity.com/paper/1810.12440/) (2018)
- [NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization](https://mlsanity.com/paper/2001.03360/) (2020)
- [Learning To Count Everything](https://mlsanity.com/paper/2104.08391/) (2021)
- [JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method](https://mlsanity.com/paper/2004.03597/) (2020)
- [T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy](https://mlsanity.com/paper/2403.14610/) (2024)
- [Few-shot Object Counting and Detection](https://mlsanity.com/paper/2207.10988/) (2022)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [VLind-Bench: Measuring Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2406.08702/) (2024)
- [Learning to Count Objects in Natural Images for Visual Question Answering](https://mlsanity.com/paper/1802.05766/) (2018)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [CountQA: How Well Do MLLMs Count in the Wild?](https://mlsanity.com/paper/2508.06585/) (2025)
- [Small Object Detection: A Comprehensive Survey on Challenges, Techniques and Real-World Applications](https://mlsanity.com/paper/2503.20516/) (2025)
- [AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs](https://mlsanity.com/paper/2506.05328/) (2025)
- [MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs](https://mlsanity.com/paper/2511.14159/) (2025)
- [LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2502.12359/) (2025)
- [Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding](https://mlsanity.com/paper/2509.23050/) (2025)

## Built on by (most cited first, 1 of 1)

- [Thinking with Anchors: Grounded and Efficient Document Reasoning](https://mlsanity.com/paper/2608.04424/) (2026)

## Abstract (arXiv)

Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.