# Visual Enumeration Remains Challenging for Multimodal Generative AI arXiv: https://arxiv.org/abs/2402.03328 ยท HTML: https://mlsanity.com/paper/2402.03328/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 29 of its 82 references that are in the map. - First posted on arXiv 9 Jan 2024. ## Builds on (most cited first, 29 of 29) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023) - [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022) - [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023) - [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021) - [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021) - [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023) - [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023) - [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022) - [Using cognitive psychology to understand GPT-3](https://mlsanity.com/paper/2206.14576/) (2022) - [Multimodal Foundation Models: From Specialists to General-Purpose Assistants](https://mlsanity.com/paper/2309.10020/) (2023) - [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023) - [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021) - [Multimodal Image Synthesis and Editing: The Generative AI Era](https://mlsanity.com/paper/2112.13592/) (2021) - [Learning to Count Objects in Natural Images for Visual Question Answering](https://mlsanity.com/paper/1802.05766/) (2018) - [Point, Segment and Count: A Generalized Framework for Object Counting](https://mlsanity.com/paper/2311.12386/) (2023) - [Counting Everyday Objects in Everyday Scenes](https://mlsanity.com/paper/1604.03505/) (2016) - [Training-free Object Counting with Prompts](https://mlsanity.com/paper/2307.00038/) (2023) - [Counting Guidance for High Fidelity Text-to-Image Synthesis](https://mlsanity.com/paper/2306.17567/) (2023) - [Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning models](https://mlsanity.com/paper/2303.07735/) (2023) - [Interpretable Counting for Visual Question Answering](https://mlsanity.com/paper/1712.08697/) (2017) - [Revisiting Crowd Counting: State-of-the-art, Trends, and Future Perspectives](https://mlsanity.com/paper/2209.07271/) (2022) ## Built on by (most cited first, 5 of 5) - [Sequential Enumeration in Large Language Models](https://mlsanity.com/paper/2512.04727/) (2025) - [Estimating the distribution of numerosity and non-numerical visual magnitudes in natural scenes using computer vision](https://mlsanity.com/paper/2409.11028/) (2024) - [Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models](https://mlsanity.com/paper/2512.15254/) (2025) - [UniGlyph: A Seven-Segment Script for Universal Language Representation](https://mlsanity.com/paper/2410.08974/) (2024) - [NumerosityVLM: A Cognitively Inspired Benchmark for Interpreting Numerosity Representations in Vision-Language Models](https://mlsanity.com/paper/2608.15425/) (2026) ## Abstract (arXiv) Many animal species can approximately judge the number of objects in a visual scene at a single glance, and humans can further determine the exact cardinality of a set by deploying systematic counting procedures. In contrast, it has been observed that even state-of-the-art AI systems have very limited enumeration skills. In this work, we propose two benchmark tasks inspired by cognitive science that allow to precisely evaluate the visual enumeration capabilities of multimodal foundation models, thereby providing an objective measure of their number sense and counting level. We consider popular visual question answering models (BLIP, LLaVA and ViLT) as well as advanced image-to-text (Gemini, GPT and Qwen) and text-to-image (DALL-E, FLUX and Stable Diffusion) AI systems. Our analyses show that even the most advanced models cannot reliably name the number of objects in simple visual stimuli or generate images containing a target number of items, as indexed by their low accuracy in both types of tasks. Especially for numbers outside the subitizing range, their responses are often far from the target numerosity, and, in stark contrast with human behavior, in many cases the distribution of errors depends on the object category. We also observe some striking mistakes with small numbers. Our findings demonstrate that developing an intuitive visual understanding of number remains challenging for AI models and that merely increasing model size might not be a viable strategy to promote the emergence of systematic counting skills. We release the full code of our benchmark to facilitate the evaluation of enumeration skills in future AI systems.