# Zero-Shot Object Counting with Language-Vision Models

arXiv: https://arxiv.org/abs/2309.13097 · HTML: https://mlsanity.com/paper/2309.13097/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 67 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Builds on (most cited first, 30 of 49)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds](https://mlsanity.com/paper/1808.01050/) (2018)
- [f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning](https://mlsanity.com/paper/1903.10132/) (2019)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [Drone-based Object Counting by Spatially Regularized Regional Proposal Network](https://mlsanity.com/paper/1707.05972/) (2017)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization](https://mlsanity.com/paper/2001.03360/) (2020)
- [Context-Aware Crowd Counting](https://mlsanity.com/paper/1811.10452/) (2018)
- [Learning To Count Everything](https://mlsanity.com/paper/2104.08391/) (2021)
- [Distribution Matching for Crowd Counting](https://mlsanity.com/paper/2009.13077/) (2020)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [A Large Contextual Dataset for Classification, Detection and Counting of Cars with Deep Learning](https://mlsanity.com/paper/1609.04453/) (2016)
- [Class-Agnostic Counting](https://mlsanity.com/paper/1811.00472/) (2018)
- [Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Counting](https://mlsanity.com/paper/2203.08354/) (2022)
- [CounTR: Transformer-based Generalised Visual Counting](https://mlsanity.com/paper/2208.13721/) (2022)
- [Localization in the Crowd with Topological Constraints](https://mlsanity.com/paper/2012.12482/) (2020)
- [Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks](https://mlsanity.com/paper/1712.01928/) (2017)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)

## Built on by (most cited first, 9 of 9)

- [OmniCount: Multi-label Object Counting with Semantic-Geometric Priors](https://mlsanity.com/paper/2403.05435/) (2024)
- [Every Shot Counts: Using Exemplars for Repetition Counting in Videos](https://mlsanity.com/paper/2403.18074/) (2024)
- [Assessing Sample Quality via the Latent Space of Generative Models](https://mlsanity.com/paper/2407.15171/) (2024)
- [UNICBench: UNIfied Counting Benchmark for MLLM](https://mlsanity.com/paper/2603.00595/) (2026)
- [Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark](https://mlsanity.com/paper/2405.06634/) (2024)
- [What is the Right Embedding Space for Contrastive Learning in Referring Expression Counting?](https://mlsanity.com/paper/2505.22850/) (2025)
- [SITUATE -- Synthetic Object Counting Dataset for VLM training](https://mlsanity.com/paper/2602.00108/) (2026)
- [Expanding Zero-Shot Object Counting with Rich Prompts](https://mlsanity.com/paper/2505.15398/) (2025)
- [Train the Agent, Not the Expert: Learning to Harness Heterogeneous Experts for Multi-Turn Visual Reasoning](https://mlsanity.com/paper/2605.29894/) (2026)

## Abstract (arXiv)

Class-agnostic object counting aims to count object instances of an arbitrary class at test time. It is challenging but also enables many potential applications. Current methods require human-annotated exemplars as inputs which are often unavailable for novel categories, especially for autonomous systems. Thus, we propose zero-shot object counting (ZSC), a new setting where only the class name is available during test time. This obviates the need for human annotators and enables automated operation. To perform ZSC, we propose finding a few object crops from the input image and use them as counting exemplars. The goal is to identify patches containing the objects of interest while also being visually representative for all instances in the image. To do this, we first construct class prototypes using large language-vision models, including CLIP and Stable Diffusion, to select the patches containing the target objects. Furthermore, we propose a ranking model that estimates the counting error of each patch to select the most suitable exemplars for counting. Experimental results on a recent class-agnostic counting dataset, FSC-147, validate the effectiveness of our method.