# CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model

arXiv: https://arxiv.org/abs/2304.04231 · HTML: https://mlsanity.com/paper/2304.04231/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 51 of its 66 references that are in the map.
- First posted on arXiv 9 Apr 2023.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes](https://mlsanity.com/paper/1802.10062/) (2018)
- [A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model](https://mlsanity.com/paper/2112.14757/) (2021)
- [Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowds](https://mlsanity.com/paper/1808.01050/) (2018)
- [Bayesian Loss for Crowd Count Estimation with Point Supervision](https://mlsanity.com/paper/1908.03684/) (2019)
- [Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework](https://mlsanity.com/paper/2107.12746/) (2021)
- [NWPU-Crowd: A Large-Scale Benchmark for Crowd Counting and Localization](https://mlsanity.com/paper/2001.03360/) (2020)
- [JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method](https://mlsanity.com/paper/2004.03597/) (2020)
- [Switching Convolutional Neural Network for Crowd Counting](https://mlsanity.com/paper/1708.00199/) (2017)
- [CLIP2StyleGAN: Unsupervised Extraction of StyleGAN Edit Directions](https://mlsanity.com/paper/2112.05219/) (2021)
- [An End-to-End Transformer Model for Crowd Localization](https://mlsanity.com/paper/2202.13065/) (2022)
- [Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detection](https://mlsanity.com/paper/1906.07538/) (2019)
- [Boosting Crowd Counting via Multifaceted Attention](https://mlsanity.com/paper/2203.02636/) (2022)
- [Localization in the Crowd with Topological Constraints](https://mlsanity.com/paper/2012.12482/) (2020)
- [Point in, Box out: Beyond Counting Persons in Crowds](https://mlsanity.com/paper/1904.01333/) (2019)
- [TransCrowd: weakly-supervised crowd counting with transformers](https://mlsanity.com/paper/2104.09116/) (2021)
- [Leveraging Unlabeled Data for Crowd Counting by Learning to Rank](https://mlsanity.com/paper/1803.03095/) (2018)
- [AvatarCLIP: Zero-Shot Text-Driven Generation and Animation of 3D Avatars](https://mlsanity.com/paper/2205.08535/) (2022)
- [Exploiting Unlabeled Data in CNNs by Self-supervised Learning to Rank](https://mlsanity.com/paper/1902.06285/) (2019)
- [From Open Set to Closed Set: Counting Objects by Spatial Divide-and-Conquer](https://mlsanity.com/paper/1908.06473/) (2019)
- [OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression](https://mlsanity.com/paper/2206.02338/) (2022)
- [ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cues](https://mlsanity.com/paper/2201.06696/) (2022)
- [CNN-based Cascaded Multi-task Learning of High-level Prior and Density Estimation for Crowd Counting](https://mlsanity.com/paper/1707.09605/) (2017)
- [Spatial Uncertainty-Aware Semi-Supervised Crowd Counting](https://mlsanity.com/paper/2107.13271/) (2021)
- [A Self-Training Approach for Point-Supervised Object Detection and Counting in Crowds](https://mlsanity.com/paper/2007.12831/) (2020)

## Built on by (most cited first, 30 of 47)

- [CLIP-Count: Towards Text-Guided Zero-Shot Object Counting](https://mlsanity.com/paper/2305.07304/) (2023)
- [SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model](https://mlsanity.com/paper/2306.02245/) (2023)
- [KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models](https://mlsanity.com/paper/2407.17773/) (2024)
- [Teach CLIP to Develop a Number Sense for Ordinal Regression](https://mlsanity.com/paper/2408.03574/) (2024)
- [OmniCount: Multi-label Object Counting with Semantic-Geometric Priors](https://mlsanity.com/paper/2403.05435/) (2024)
- [A Unified Framework for 3D Scene Understanding](https://mlsanity.com/paper/2407.03263/) (2024)
- [CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Reward](https://mlsanity.com/paper/2504.03724/) (2025)
- [AFreeCA: Annotation-Free Counting for All](https://mlsanity.com/paper/2403.04943/) (2024)
- [CLIP-EBC: CLIP Can Count Accurately through Enhanced Blockwise Classification](https://mlsanity.com/paper/2403.09281/) (2024)
- [LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition](https://mlsanity.com/paper/2309.12780/) (2023)
- [Single Domain Generalization for Few-Shot Counting via Universal Representation Matching](https://mlsanity.com/paper/2505.16778/) (2025)
- [Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2510.09320/) (2025)
- [Learning to Count without Annotations](https://mlsanity.com/paper/2307.08727/) (2023)
- [SQLNet: Scale-Modulated Query and Localization Network for Few-Shot Class-Agnostic Counting](https://mlsanity.com/paper/2311.10011/) (2023)
- [Learning Spatial Similarity Distribution for Few-shot Object Counting](https://mlsanity.com/paper/2405.11770/) (2024)
- [Robust Zero-Shot Crowd Counting and Localization With Adaptive Resolution SAM](https://mlsanity.com/paper/2402.17514/) (2024)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [Early Estimation of Language to Latent Alignment in Diffusion Models](https://mlsanity.com/paper/2512.08505/) (2025)
- [CoLA: Conditional Dropout and Language-driven Robust Dual-modal Salient Object Detection](https://mlsanity.com/paper/2407.06780/) (2024)
- [Shifted Autoencoders for Point Annotation Restoration in Object Counting](https://mlsanity.com/paper/2312.07190/) (2023)
- [Dynamic Proxy Domain Generalizes the Crowd Localization by Better Binary Segmentation](https://mlsanity.com/paper/2404.13992/) (2024)
- [TFCounter:Polishing Gems for Training-Free Object Counting](https://mlsanity.com/paper/2405.02301/) (2024)
- [Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports](https://mlsanity.com/paper/2405.14230/) (2024)
- [Ranking-aware adapter for text-driven image ordering with CLIP](https://mlsanity.com/paper/2412.06760/) (2024)
- [STAR: A Foundation Model-driven Framework for Robust Task Planning and Failure Recovery in Robotic Systems](https://mlsanity.com/paper/2503.06060/) (2025)
- [FiGO: Fine-Grained Object Counting without Annotations](https://mlsanity.com/paper/2504.11705/) (2025)
- [Parameterized Diffusion Optimization enabled Autoregressive Ordinal Regression for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2507.04978/) (2025)
- [Text-promptable Object Counting via Quantity Awareness Enhancement](https://mlsanity.com/paper/2507.06679/) (2025)
- [Semi-Supervised Multi-View Crowd Counting by Ranking Multi-View Fusion Models](https://mlsanity.com/paper/2512.16243/) (2025)
- [Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting](https://mlsanity.com/paper/2602.12774/) (2026)

## Abstract (arXiv)

Supervised crowd counting relies heavily on costly manual labeling, which is difficult and expensive, especially in dense scenes. To alleviate the problem, we propose a novel unsupervised framework for crowd counting, named CrowdCLIP. The core idea is built on two observations: 1) the recent contrastive pre-trained vision-language model (CLIP) has presented impressive performance on various downstream tasks; 2) there is a natural mapping between crowd patches and count text. To the best of our knowledge, CrowdCLIP is the first to investigate the vision language knowledge to solve the counting problem. Specifically, in the training stage, we exploit the multi-modal ranking loss by constructing ranking text prompts to match the size-sorted crowd patches to guide the image encoder learning. In the testing stage, to deal with the diversity of image patches, we propose a simple yet effective progressive filtering strategy to first select the highly potential crowd patches and then map them into the language space with various counting intervals. Extensive experiments on five challenging datasets demonstrate that the proposed CrowdCLIP achieves superior performance compared to previous unsupervised state-of-the-art counting methods. Notably, CrowdCLIP even surpasses some popular fully-supervised methods under the cross-dataset setting. The source code will be available at https://github.com/dk-liang/CrowdCLIP.