mlsanityData to 7 Oct 2026

Understanding Counting Mechanisms in Large Language and Vision-Language Models

arXiv 2511.17699 ↗First posted 21 Nov 2025cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Counting is one of the fundamental abilities of large language models (LLMs) and large vision-language models (LVLMs). This paper examines how these foundation models represent and compute numerical information in counting tasks. We use controlled experiments with repeated textual and visual items and analyze counting in LLMs and LVLMs through a set of behavioral, observational, and causal mediation analyses. To this end, we design a specialized tool, CountScope, for the mechanistic interpretability of numerical content. Results show that individual tokens or visual features encode latent positional count information that can be extracted and transferred across contexts. Layerwise analyses reveal a progressive emergence of numerical representations, with lower layers encoding small counts and higher layers representing larger ones. We identify an internal counter mechanism that updates with each item, stored mainly in the final token or region. In LVLMs, numerical information also appears in visual embeddings, shifting between background and foreground regions depending on spatial composition. We further reveal that models rely on structural cues such as separators in text, which act as shortcuts for tracking item counts and strongly influence the accuracy of numerical predictions. Overall, counting emerges as a structured, layerwise process in LLMs and follows the same general pattern in LVLMs, shaped by the properties of the vision encoder.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 30 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

24 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting6 Oct
  2. 2025Uncovering Grounding IDs: How External Cues Shape Multimodal Binding28 Sep
  3. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct
  4. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  5. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  6. 2023Teaching CLIP to Count to Ten23 Feb

Other references · 18

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  9. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  10. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  11. 2024How to use and interpret activation patchingcited by 191
  12. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
6 more
  1. 2024Why Do Large Language Models (LLMs) Struggle to Count Letters?cited by 37
  2. 2024Counting Ability of Large Language Models and Impact of Tokenizationcited by 29
  3. 2025Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMscited by 14
  4. 2024Can We Count on LLMs? The Fixed-Effect Fallacy and Claims of GPT-4 Capabilitiescited by 13
  5. 2023Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Modelscited by 10
  6. 2024Contextual Counting: A Mechanistic Study of Transformers on a Quantitative Taskcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Counting Circuits: Mechanistic Interpretability of Visual Reasoning in Large Vision-Language Modelscited by 3
  2. 2026CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Modelscited by 1
  3. 2026Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challengescited by 0
  4. 2026Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Unveiling the Visual Counting Bottleneck in Vision-Language Models
  2. 2026HoloCount: A Holistic Visual Counting Benchmark for MLLMs
  3. 2026Counting to Four is still a Chore for VLMs
  4. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
  5. 2026The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs
  6. 2025[De|Re]constructing VLMs' Reasoning in Counting
  7. 2025Can Current AI Models Count What We Mean, Not What They See? A Benchmark and Systematic Evaluation
  8. 2025Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models