mlsanityData to 13 Oct 2026

The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping

arXiv 2608.06361 ↗First posted 6 Aug 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Real-world video benchmarks provide broad coverage, but their fixed clips entangle event count, rate, duration, and visual complexity, making failure modes hard to isolate. While existing programmatic benchmarks offer better control, they score only the final answer rather than auditing reported events against executable ground truth. To bridge this gap, we introduce trace-grounded parametric profiling for event counting in three controlled video tasks: bouncing-ball wall contacts, visual blinks, and categorical state transitions. Across 2,190 videos, we vary event count N and frequency F while holding rendering fixed. Each video includes an executable event trace for capability-surface estimation and timestamp-level evaluation. Our results reveal a staged temporal failure. At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz, yet demonstrates no reliable positive-count region for transient blinking events. Thus, event representation dictates whether a model initially accesses evidence -- a limitation that compounds as count and frequency increase. In the high-count, high-frequency regime, only 0.2% of final counts are correct and the model recovers just 18.1% of true events. To test if visual access is the primary bottleneck, we increase sampling rate. Although this boosts Bounce Ball accuracy from 19.6% to 29.3%, the reported sequence agrees with ground truth only 3.7% of the time. Extra frames can therefore inflate final scores without producing faithful event recovery. Different prompting strategies yield similarly limited gains, and real-world video evaluations show the same concentration of success at low event counts. Ultimately, trace-grounded profiling shifts video evaluation from aggregate accuracy metrics to a detailed diagnostic of where temporal reasoning fails.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

26 of 32 references.

References · 26

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,595
  3. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,450
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  5. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 723
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 672
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 653
  8. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 354
  9. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 300
  10. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 229
  11. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 149
  12. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 135
14 more
  1. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  2. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  3. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  4. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  5. 2025OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalizationcited by 49
  6. 2022TransRAC: Encoding Multi-scale Temporal Correlation with Transformers for Repetitive Action Countingcited by 43
  7. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  8. 2024Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMscited by 35
  9. 2025Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoningcited by 30
  10. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?cited by 19
  11. 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?cited by 18
  12. 2024VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Modelscited by 10
  13. 2025Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmarkcited by 10
  14. 2025MORSE-500: A Programmatically Controllable Video Benchmark to Stress-Test Multimodal Reasoningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  2. 2026Benchmarking Visual State Tracking in Multimodal Video Understanding
  3. 2025CycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
  4. 2026TAU-Bench: From Anomaly Instance Tracking to Fine-Grained Video Anomaly Understanding
  5. 2025CueBench: Advancing Unified Understanding of Context-Aware Video Anomalies in Real-World
  6. 2026APT: Atomic Physical Transitions for Causal Video-Language Understanding
  7. 2026Foresight at the Event Boundary: Evaluating Physical Prediction in Video World Models
  8. 2026MBench: A Comprehensive Benchmark on Memory Capability for Video World Models