# RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark

arXiv: https://arxiv.org/abs/2610.18844 · HTML: https://mlsanity.com/paper/2610.18844/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 42 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 30 of 35)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations](https://mlsanity.com/paper/2410.02707/) (2024)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Cosmos-Reason1: From Physical Common Sense To Embodied Reasoning](https://mlsanity.com/paper/2503.15558/) (2025)
- [TallyQA: Answering Complex Counting Questions](https://mlsanity.com/paper/1810.12440/) (2018)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Scaling RL to Long Videos](https://mlsanity.com/paper/2507.07966/) (2025)
- [TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models](https://mlsanity.com/paper/2410.23266/) (2024)
- [Counting Out Time: Class Agnostic Video Repetition Counting in the Wild](https://mlsanity.com/paper/2006.15418/) (2020)
- [VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception](https://mlsanity.com/paper/2509.21100/) (2025)
- [TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting](https://mlsanity.com/paper/2204.01018/) (2022)
- [Lost in Time: A New Temporal Benchmark for VideoLLMs](https://mlsanity.com/paper/2410.07752/) (2024)
- [Temporal Reasoning Transfer from Text to Video](https://mlsanity.com/paper/2410.06166/) (2024)
- [Context-aware and Scale-insensitive Temporal Repetition Counting](https://mlsanity.com/paper/2005.08465/) (2020)
- [The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?](https://mlsanity.com/paper/2403.09037/) (2024)
- [Language Models Encode the Value of Numbers Linearly](https://mlsanity.com/paper/2401.03735/) (2024)

## Abstract (arXiv)

Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently, they fail to reliably count how many times an action repeats. To investigate this limitation, we introduce RepProbe, a benchmark of 42K real and synthetic video clips containing exactly k in [1, 20] complete repetitions, with controls for clip position, temporal coverage, and duration. These controls allow us to study the counting failure directly and locate where in the model it arises. Counting accuracy collapses at high counts across the video language models we evaluate, and controlled experiments show this collapse is not explained by frame sampling alone. We compare model outputs with linear probes trained on the final-layer representations used to generate those outputs. We find a consistent representation-to-output dissociation: substantial count information remains linearly decodable even when models fail to report the correct count. Further analysis shows this failure arises primarily at the output stage, where answers concentrate on a small set of preferred numbers rather than the available count information. The dissociation holds on both real and synthetic videos and is not explained by clip duration, position, or scene content. These results indicate that models encode repetition structure but do not reliably report it in their outputs. Project page: https://arenaa.github.io/RepProbe-webpage/