# Mind the Prompt: A Novel Benchmark for Prompt-based Class-Agnostic Counting

arXiv: https://arxiv.org/abs/2409.15953 · HTML: https://mlsanity.com/paper/2409.15953/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Context-Aware Crowd Counting](https://mlsanity.com/paper/1811.10452/) (2018)
- [Learning To Count Everything](https://mlsanity.com/paper/2104.08391/) (2021)
- [Switching Convolutional Neural Network for Crowd Counting](https://mlsanity.com/paper/1708.00199/) (2017)
- [CLIP-Count: Towards Text-Guided Zero-Shot Object Counting](https://mlsanity.com/paper/2305.07304/) (2023)
- [CounTR: Transformer-based Generalised Visual Counting](https://mlsanity.com/paper/2208.13721/) (2022)
- [A Low-Shot Object Counting Network With Iterative Prototype Adaptation](https://mlsanity.com/paper/2211.08217/) (2022)
- [Few-shot Object Counting with Similarity-Aware Feature Enhancement](https://mlsanity.com/paper/2201.08959/) (2022)
- [Zero-shot Object Counting](https://mlsanity.com/paper/2303.02001/) (2023)
- [DAVE -- A Detect-and-Verify Paradigm for Low-Shot Counting](https://mlsanity.com/paper/2404.16622/) (2024)
- [VLCounter: Text-aware Visual Representation for Zero-Shot Object Counting](https://mlsanity.com/paper/2312.16580/) (2023)
- [Open-world Text-specified Object Counting](https://mlsanity.com/paper/2306.01851/) (2023)
- [Training-free Object Counting with Prompts](https://mlsanity.com/paper/2307.00038/) (2023)
- [Count-ception: Counting by Fully Convolutional Redundant Counting](https://mlsanity.com/paper/1703.08710/) (2017)
- [FCN-rLSTM: Deep Spatio-Temporal Neural Networks for Vehicle Counting in City Cameras](https://mlsanity.com/paper/1707.09476/) (2017)
- [Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting](https://mlsanity.com/paper/2305.04440/) (2023)
- [Crowd Counting Using Scale-Aware Attention Networks](https://mlsanity.com/paper/1903.02025/) (2019)
- [OmniCount: Multi-label Object Counting with Semantic-Geometric Priors](https://mlsanity.com/paper/2403.05435/) (2024)
- [ABC Easy as 123: A Blind Counter for Exemplar-Free Multi-Class Class-agnostic Counting](https://mlsanity.com/paper/2309.04820/) (2023)
- [A Fixed-Point Approach to Unified Prompt-Based Counting](https://mlsanity.com/paper/2403.10236/) (2024)
- [Multi-Camera Vehicle Counting Using Edge-AI](https://mlsanity.com/paper/2106.02842/) (2021)
- [Learning Spatial Similarity Distribution for Few-shot Object Counting](https://mlsanity.com/paper/2405.11770/) (2024)

## Built on by (most cited first, 11 of 11)

- [CountGD++: Generalized Prompting for Open-World Counting](https://mlsanity.com/paper/2512.23351/) (2025)
- [A Survey on Class-Agnostic Counting: Advancements from Reference-Based to Open-World Text-Guided Approaches](https://mlsanity.com/paper/2501.19184/) (2025)
- [Can Current AI Models Count What We Mean, Not What They See? A Benchmark and Systematic Evaluation](https://mlsanity.com/paper/2509.13939/) (2025)
- [CountingDINO: A Training-free Pipeline for Class-Agnostic Counting using Unsupervised Backbones](https://mlsanity.com/paper/2504.16570/) (2025)
- [Generalized-Scale Object Counting with Gradual Query Aggregation](https://mlsanity.com/paper/2511.08048/) (2025)
- [Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting](https://mlsanity.com/paper/2605.02752/) (2026)
- [Count Anything at Any Granularity](https://mlsanity.com/paper/2605.10887/) (2026)
- [AdaCount: Training-Free Similarity-Guided Spatial and Feature Adaptation for Zero-Shot Object Counting](https://mlsanity.com/paper/2607.02139/) (2026)
- [Getting the Numbers Rightunicodex2014Modelling Multi-Class Object Counting in Dense and Varied Scenes](https://mlsanity.com/paper/2510.02213/) (2025)
- [RECOUNT: Reference-guided Counting with Synthetic Visual Exemplars](https://mlsanity.com/paper/2608.20621/) (2026)
- [Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges](https://mlsanity.com/paper/2608.23845/) (2026)

## Abstract (arXiv)

Recently, object counting has shifted towards class-agnostic counting (CAC), which counts instances of arbitrary object classes never seen during model training. With advancements in robust vision-and-language foundation models, there is a growing interest in prompt-based CAC, where object categories are specified using natural language. However, we identify significant limitations in current benchmarks for evaluating this task, which hinder both accurate assessment and the development of more effective solutions. Specifically, we argue that the current evaluation protocols do not measure the ability of the model to understand which object has to be counted. This is due to two main factors: (i) the shortcomings of CAC datasets, which primarily consist of images containing objects from a single class, and (ii) the limitations of current counting performance evaluators, which are based on traditional class-specific counting and focus solely on counting errors. To fill this gap, we introduce the Prompt-Aware Counting (PrACo) benchmark. It comprises two targeted tests coupled with evaluation metrics specifically designed to quantitatively measure the robustness and trustworthiness of existing prompt-based CAC models. We evaluate state-of-the-art methods and demonstrate that, although some achieve impressive results on standard class-specific counting metrics, they exhibit a significant deficiency in understanding the input prompt, indicating the need for more careful training procedures or revised designs. The code for reproducing our results is available at https://github.com/ciampluca/PrACo.