# Can Current AI Models Count What We Mean, Not What They See? A Benchmark and Systematic Evaluation

arXiv: https://arxiv.org/abs/2509.13939 · HTML: https://mlsanity.com/paper/2509.13939/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 57 references that are in the map.
- First posted on arXiv 17 Sep 2025.

## Builds on (most cited first, 30 of 45)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [Drone-based Object Counting by Spatially Regularized Regional Proposal Network](https://mlsanity.com/paper/1707.05972/) (2017)
- [Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework](https://mlsanity.com/paper/2107.12746/) (2021)
- [Learning To Count Everything](https://mlsanity.com/paper/2104.08391/) (2021)
- [Distribution Matching for Crowd Counting](https://mlsanity.com/paper/2009.13077/) (2020)
- [JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method](https://mlsanity.com/paper/2004.03597/) (2020)
- [Class-Agnostic Counting](https://mlsanity.com/paper/1811.00472/) (2018)
- [CLIP-Count: Towards Text-Guided Zero-Shot Object Counting](https://mlsanity.com/paper/2305.07304/) (2023)
- [Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Counting](https://mlsanity.com/paper/2203.08354/) (2022)
- [Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detection](https://mlsanity.com/paper/1906.07538/) (2019)
- [Localization in the Crowd with Topological Constraints](https://mlsanity.com/paper/2012.12482/) (2020)
- [A Low-Shot Object Counting Network With Iterative Prototype Adaptation](https://mlsanity.com/paper/2211.08217/) (2022)
- [Point in, Box out: Beyond Counting Persons in Crowds](https://mlsanity.com/paper/1904.01333/) (2019)
- [Few-shot Object Counting with Similarity-Aware Feature Enhancement](https://mlsanity.com/paper/2201.08959/) (2022)
- [CountGD: Multi-Modal Open-World Counting](https://mlsanity.com/paper/2407.04619/) (2024)
- [Zero-shot Object Counting](https://mlsanity.com/paper/2303.02001/) (2023)
- [LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models](https://mlsanity.com/paper/2501.18954/) (2025)
- [Few-shot Object Counting and Detection](https://mlsanity.com/paper/2207.10988/) (2022)
- [Where are the Blobs: Counting by Localization with Point Supervision](https://mlsanity.com/paper/1807.09856/) (2018)
- [DAVE -- A Detect-and-Verify Paradigm for Low-Shot Counting](https://mlsanity.com/paper/2404.16622/) (2024)
- [Iterative Crowd Counting](https://mlsanity.com/paper/1807.09959/) (2018)
- [VLCounter: Text-aware Visual Representation for Zero-Shot Object Counting](https://mlsanity.com/paper/2312.16580/) (2023)
- [Open-world Text-specified Object Counting](https://mlsanity.com/paper/2306.01851/) (2023)
- [Point, Segment and Count: A Generalized Framework for Object Counting](https://mlsanity.com/paper/2311.12386/) (2023)

## Built on by (most cited first, 7 of 7)

- [CountGD++: Generalized Prompting for Open-World Counting](https://mlsanity.com/paper/2512.23351/) (2025)
- [CountEx: Fine-Grained Counting via Exemplars and Exclusion](https://mlsanity.com/paper/2602.19432/) (2026)
- [Count Anything at Any Granularity](https://mlsanity.com/paper/2605.10887/) (2026)
- [The MixCount Dataset: Bridging the Data Gap for Open-Vocabulary Object Counting](https://mlsanity.com/paper/2605.18063/) (2026)
- [RECOUNT: Reference-guided Counting with Synthetic Visual Exemplars](https://mlsanity.com/paper/2608.20621/) (2026)
- [Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges](https://mlsanity.com/paper/2608.23845/) (2026)
- [Counting Beyond Instances: A Benchmark for Group-Individual Object Counting](https://mlsanity.com/paper/2609.04716/) (2026)

## Abstract (arXiv)

Visual counting is a fundamental yet challenging task, especially when users need to count objects of a specific type in complex scenes. While recent models, including class-agnostic counting models and large vision-language models (VLMs), show promise in counting tasks, their ability to perform fine-grained, intent-driven counting remains unclear. In this paper, we introduce PairTally, a benchmark dataset specifically designed to evaluate fine-grained visual counting. Each of the 681 high-resolution images in PairTally contains two object categories, requiring models to distinguish and count based on subtle differences in shape, size, color, or semantics. The dataset includes both inter-category (distinct categories) and intra-category (closely related subcategories) settings, making it suitable for rigorous evaluation of selective counting capabilities. We benchmark a variety of state-of-the-art models, including exemplar-based methods, language-prompted models, and large VLMs. Our results show that despite recent advances, current models struggle to reliably count what users intend, especially in fine-grained and visually ambiguous cases. PairTally provides a new foundation for diagnosing and improving fine-grained visual counting systems.