mlsanityData to 14 Oct 2026

CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

arXiv 2605.17826 ↗First posted 18 May 2026cs.CV cs.AIcited by 1

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 57 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

49 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Models19 Mar
  2. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blind21 Feb
  3. 2024Probing Visual Language Priors in VLMs31 Dec
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 44

  1. 2023Segment Anythingcited by 8,234
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  8. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  11. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  12. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
32 more
  1. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  2. 2023Teaching CLIP to Count to Tencited by 198
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 179
  4. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  5. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  6. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  7. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  8. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problemcited by 75
  9. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 55
  10. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  11. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  12. 2024VLind-Bench: Measuring Language Priors in Large Vision-Language Modelscited by 45
  13. 2023Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Modelscited by 43
  14. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  15. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfactscited by 32
  16. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  17. 2024LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Modelscited by 20
  18. 2024Ask LLMs Directly, "What shapes your bias?": Measuring Social Bias in Large Language Modelscited by 14
  19. 2024The Illusion-Illusion: Vision Language Models See Illusions Where There Are Nonecited by 14
  20. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?cited by 14
  21. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  22. 2024BiasDora: Exploring Hidden Biased Associations in Vision-Language Modelscited by 12
  23. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  24. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 9
  25. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMscited by 8
  26. 2024DARE: Diverse Visual Question Answering with Robustness Evaluationcited by 6
  27. 2025B-score: Detecting biases in large language models using response historycited by 6
  28. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventionscited by 6
  29. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Modelscited by 5
  30. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 4
  31. 2025Understanding Counting Mechanisms in Large Language and Vision-Language Modelscited by 4
  32. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Counting to Four is still a Chore for VLMs
  2. 2026HoloCount: A Holistic Visual Counting Benchmark for MLLMs
  3. 2025Vision Language Models are Biased
  4. 2026The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs
  5. 2025CF-VLM:CounterFactual Vision-Language Fine-tuning
  6. 2026Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models
  7. 2025[De|Re]constructing VLMs' Reasoning in Counting
  8. 2025CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models