mlsanityData to 7 Oct 2026

PushupBench: Your VLM is not good at counting pushups

arXiv 2604.23407 ↗First posted 25 Apr 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (VLMs) can recognize textitwhat happens in video but fail to count textithow many times. We introduce PushupBench, 446 long-form clips (avg. 36.7s) for evaluating repetition counting. The best frontier model achieves 42.1% exact accuracy; open-source 4B models score ~6%, matching supervised baselines. We show that accuracy alone misleads -- weaker models exploit the modal count rather than reason temporally. Fine-tuning on counting with 1k samples transfers to general video understanding: MVBench (+2.15), PerceptionTest (+1.88), TVBench (+4.54), suggesting counting is a proxy for broader temporal reasoning.PushupBench incorporated in textttlmms-eval (https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/1262) and hosted on (pushupbench.com/)

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 13 references that are in the map. First posted on arXiv 25 Apr 2026.

Builds on

10 of 13 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions21 Nov
  2. 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting6 Oct

Other references · 8

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  3. 2020Counting Out Time: Class Agnostic Video Repetition Counting in the Wildcited by 67
  4. 2022TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Countingcited by 42
  5. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  6. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  7. 2023PoseRAC: Pose Saliency Transformer for Repetitive Action Countingcited by 17
  8. 2024OVR: A Dataset for Open Vocabulary Temporal Repetition Counting in Videoscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Count Is There, but Misaligned: Understanding and Correcting Counting Failures in VLMs
  2. 2025[De|Re]constructing VLMs' Reasoning in Counting
  3. 2025AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs
  4. 2025Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models
  5. 2026Counting to Four is still a Chore for VLMs
  6. 2023Teaching CLIP to Count to Ten
  7. 2026Diagnosing Long-Video Quantitative Reasoning in Multimodal LLMs via Enumeration and Counting
  8. 2026UNICBench: UNIfied Counting Benchmark for MLLM