mlsanityData to 7 Oct 2026

Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting

arXiv 2510.04401 ↗First posted 6 Oct 2025cs.CV cs.AIcited by 17

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) have become a central focus of today's AI community, owing to their impressive abilities gained from training on large-scale vision-language data from the Web. These models have demonstrated strong performance across diverse tasks, including image understanding, video understanding, complex visual reasoning, and embodied AI. Despite these noteworthy successes, a fundamental question remains: Can VLMs count objects correctly? In this paper, we introduce a simple yet effective benchmark, VLMCountBench, designed under a minimalist setting with only basic geometric shapes (e.g., triangles, circles) and their compositions, focusing exclusively on counting tasks without interference from other factors. We adopt strict independent variable control and systematically study the effects of simple properties such as color, size, and prompt refinement in a controlled ablation. Our empirical results reveal that while VLMs can count reliably when only one shape type is present, they exhibit substantial failures when multiple shape types are combined (i.e., compositional counting). This highlights a fundamental empirical limitation of current VLMs and motivates important directions for future research.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 81 of its 92 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

81 of 92 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Kimi-VL Technical Report10 Apr
  2. 2024VHELM: A Holistic Evaluation of Vision Language Models9 Oct
  3. 2024PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns20 Mar
  4. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models15 Mar
  5. 2023CLIP-Count: Towards Text-Guided Zero-Shot Object Counting12 May
  6. 2023Teaching CLIP to Count to Ten23 Feb
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 74

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2019Towards VQA Models That Can Readcited by 1,696
38 more
  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  3. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  5. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  6. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  7. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  8. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 297
  9. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  10. 2022Transformers Learn Shortcuts to Automatacited by 260
  11. 2024Chain of Thought Empowers Transformers to Solve Inherently Serial Problemscited by 256
  12. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  13. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  14. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  15. 2023Fast Attention Requires Bounded Entriescited by 102
  16. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  17. 2023GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planningcited by 61
  18. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  19. 2023Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answeringcited by 51
  20. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 49
  21. 2024Make It Count: Text-to-Image Generation with an Accurate Number of Objectscited by 47
  22. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  23. 2024Why Larger Language Models Do In-context Learning Differently?cited by 44
  24. 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41
  25. 2024On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysiscited by 41
  26. 2024On Statistical Rates and Provably Efficient Criteria of Latent Diffusion Transformers (DiTs)cited by 39
  27. 2025T2VPhysBench: A First-Principles Benchmark for Physical Consistency in Text-to-Video Generationcited by 35
  28. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  29. 2024The Fine-Grained Complexity of Gradient Computation for Training Large Language Modelscited by 30
  30. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  31. 2023The Descriptive Complexity of Graph Neural Networkscited by 26
  32. 2024Circuit Complexity Bounds for RoPE-based Transformer Architecturecited by 26
  33. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMscited by 21
  34. 2024Fundamental Limits of Prompt Tuning Transformers: Universality, Capacity and Efficiencycited by 21
  35. 2024On Statistical Rates of Conditional Diffusion Transformers: Approximation, Estimation and Minimax Optimalitycited by 21
  36. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  37. 2025T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generationcited by 18
  38. 2025On Computational Limits and Provably Efficient Criteria of Visual Autoregressive Models: A Fine-Grained Complexity Analysiscited by 14

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 14.

17 papers.

Sort:

By area:

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventionscited by 6
  3. 2025Understanding Counting Mechanisms in Large Language and Vision-Language Modelscited by 4
  4. 2026Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Modelscited by 3
  5. 2026SITUATE -- Synthetic Object Counting Dataset for VLM trainingcited by 2
  6. 2026Next-Gen CAPTCHAs: Leveraging the Cognitive Gap for Scalable and Diverse GUI-Agent Defensecited by 2
  7. 2026WeaveLA: Event Driven Cross-Subtask Latent Memory Weaving for Repetitive Robot Manipulationcited by 2
  8. 2026Can Vision Language Models Learn Intuitive Physics from Interaction?cited by 1
  9. 2026Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Familiescited by 1
  10. 2026GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinationscited by 1
  11. 2026Counting to Four is still a Chore for VLMscited by 1
  12. 2026Unveiling the Visual Counting Bottleneck in Vision-Language Modelscited by 1
5 more
  1. 2026TriViewBench: Controlled Complexity Scaling for Multi-View Structural Reasoning in MLLMscited by 1
  2. 2026The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMscited by 1
  3. 2025Quantification and object perception in Multimodal Large Language Models and human linguistic cognitioncited by 0
  4. 2026PushupBench: Your VLM is not good at counting pushupscited by 0
  5. 2026Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025[De|Re]constructing VLMs' Reasoning in Counting
  2. 2024LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Models
  3. 2026HoloCount: A Holistic Visual Counting Benchmark for MLLMs
  4. 2025Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models
  5. 2025Can Current AI Models Count What We Mean, Not What They See? A Benchmark and Systematic Evaluation
  6. 2025Vision Language Models are Biased
  7. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  8. 2025CountQA: How Well Do MLLMs Count in the Wild?