# PushupBench: Your VLM is not good at counting pushups

arXiv: https://arxiv.org/abs/2604.23407 · HTML: https://mlsanity.com/paper/2604.23407/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 25 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 10 of 10)

- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [Counting Out Time: Class Agnostic Video Repetition Counting in the Wild](https://mlsanity.com/paper/2006.15418/) (2020)
- [TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Counting](https://mlsanity.com/paper/2204.01018/) (2022)
- [Lost in Time: A New Temporal Benchmark for VideoLLMs](https://mlsanity.com/paper/2410.07752/) (2024)
- [SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capability](https://mlsanity.com/paper/2503.13983/) (2025)
- [Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting](https://mlsanity.com/paper/2510.04401/) (2025)
- [PoseRAC: Pose Saliency Transformer for Repetitive Action Counting](https://mlsanity.com/paper/2303.08450/) (2023)
- [OVR: A Dataset for Open Vocabulary Temporal Repetition Counting in Videos](https://mlsanity.com/paper/2407.17085/) (2024)
- [Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions](https://mlsanity.com/paper/2511.17722/) (2025)

## Built on by (most cited first, 1 of 1)

- [RepProbe: Can Video Language Models Count Repeated Actions? A Diagnostic Benchmark](https://mlsanity.com/paper/2610.18844/) (2026)

## Abstract (arXiv)

Large vision-language models (VLMs) can recognize textitwhat happens in video but fail to count textithow many times. We introduce PushupBench, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1% exact accuracy; open-source 4B models score ~6%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in textttlmms-eval (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)