mlsanityData to 7 Oct 2026

TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

arXiv 2410.23266 ↗First posted 30 Oct 2024cs.CV cs.AI cs.CLcited by 71

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Existing benchmarks often highlight the remarkable performance achieved by state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal context for video understanding. However, how well do the models truly perform visual temporal reasoning? Our study of existing benchmarks shows that this capability of MFMs is likely overestimated as many questions can be solved by using a single, few, or out-of-order frames. To systematically examine current visual temporal reasoning tasks, we propose three principles with corresponding metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame Information Disparity. Following these principles, we introduce TOMATO, Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to rigorously assess MFMs' temporal reasoning capabilities in video understanding. TOMATO comprises 1,484 carefully curated, human-annotated questions spanning six tasks (i.e., action count, direction, rotation, shape & trend, velocity & frequency, and visual cues), applied to 1,417 videos, including 805 self-recorded and -generated videos, that encompass human-centric, real-world, and simulated scenarios. Our comprehensive evaluation reveals a human-model performance gap of 57.3% with the best-performing model. Moreover, our in-depth analysis uncovers more fundamental limitations beyond this gap in current MFMs. While they can accurately recognize events in isolated frames, they fail to interpret these frames as a continuous sequence. We believe TOMATO will serve as a crucial testbed for evaluating the next-generation MFMs and as a call to the community to develop AI systems capable of comprehending human world dynamics through the video modality.

Key facts. Cited by 71 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 50 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

38 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos27 Jun
  2. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Models23 May

Other references · 36

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  4. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  7. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2016Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Imagecited by 692
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  12. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
24 more
  1. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  2. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  3. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  4. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  5. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  6. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  7. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  8. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  9. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  10. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  11. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  12. 2018Actor and Observer: Joint Modeling of First and Third-Person Videoscited by 102
  13. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  14. 2024Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Modelscited by 90
  15. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  16. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  17. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  18. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  19. 2023LvBench: A Benchmark for Long-form Video Understanding with Versatile Multi-modal Question Answeringcited by 25
  20. 2021No frame left behind: Full Video Action Recognitioncited by 24
  21. 2022Can Shuffling Video Benefit Temporal Bias Problem: A Novel Training Framework for Temporal Groundingcited by 22
  22. 2022SCVRL: Shuffled Contrastive Video Representation Learningcited by 14
  23. 2016Shuffle and Learn: Unsupervised Learning using Temporal Order Verificationcited by 13
  24. 2024VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Modelcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 33; 2026: 38.

71 papers.

Sort:

By area:

  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2025Kimi-VL Technical Reportcited by 381
  3. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  4. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 104
  5. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  6. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  7. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  8. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 30
  9. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 23
  10. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  11. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 21
  12. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
38 more
  1. 2025EOC-Bench: Can MLLMs Identify, Recall, and Forecast Objects in an Egocentric World?cited by 20
  2. 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?cited by 18
  3. 2025Reasoning in Computer Vision: Taxonomy, Models, Tasks, and Methodologiescited by 17
  4. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!cited by 15
  5. 2025Fostering Video Reasoning via Next-Event Predictioncited by 15
  6. 2025VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Modelscited by 14
  7. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  8. 2025It's High Time: A Survey of Temporal Question Answeringcited by 12
  9. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  10. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 11
  11. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 10
  12. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 10
  13. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  14. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 9
  15. 2025Causality Matters: How Temporal Information Emerges in Video Language Modelscited by 9
  16. 2025Map the Flow: Revealing Hidden Pathways of Information in VideoLLMscited by 9
  17. 2026LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understandingcited by 8
  18. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  19. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  20. 2025UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?cited by 6
  21. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understandingcited by 6
  22. 2026SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenancecited by 6
  23. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 6
  24. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 6
  25. 2026MOSS-VL Technical Reportcited by 5
  26. 2025Systematic Evaluation of Large Vision-Language Models for Surgical Artificial Intelligencecited by 4
  27. 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
  28. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  29. 2025Taming Hallucinations: Boosting MLLMs' Video Understanding via Counterfactual Video Generationcited by 4
  30. 2025Re:Verse -- Can Your VLM Read a Manga?cited by 3
  31. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 3
  32. 2026YoCausal: How Far is Video Generation from World Model? A Causality Perspectivecited by 3
  33. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 3
  34. 2026CapRL++: Unified Reinforcement Learning with Verifiable Rewards for Dense Image and Video Captioningcited by 3
  35. 2025ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Planscited by 2
  36. 2025The Potential and Limitations of Vision-Language Models for Human Motion Understanding: A Case Study in Data-Driven Stroke Rehabilitationcited by 2
  37. 2026Can Vision-Language Models Solve the Shell Game?cited by 2
  38. 2026Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videoscited by 2

50 most cited of 71 All 71 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  2. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding
  3. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models
  4. 2026Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning
  5. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos
  6. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  7. 2026MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos
  8. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos