# Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!

arXiv: https://arxiv.org/abs/2501.10674 · HTML: https://mlsanity.com/paper/2501.10674/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 25 references that are in the map.
- First posted on arXiv 18 Jan 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models](https://mlsanity.com/paper/2410.10818/) (2024)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [A Survey on Benchmarks of Multimodal Large Language Models](https://mlsanity.com/paper/2408.08632/) (2024)
- [TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models](https://mlsanity.com/paper/2410.23266/) (2024)
- [A Survey on Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2408.15769/) (2024)
- [Lost in Time: A New Temporal Benchmark for VideoLLMs](https://mlsanity.com/paper/2410.07752/) (2024)
- [MIBench: Evaluating Multimodal Large Language Models over Multiple Images](https://mlsanity.com/paper/2407.15272/) (2024)
- [Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models](https://mlsanity.com/paper/2401.09861/) (2024)

## Built on by (most cited first, 15 of 15)

- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [Flattery in Motion: Benchmarking and Analyzing Sycophancy in Video-LLMs](https://mlsanity.com/paper/2506.07180/) (2025)
- [ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.19409/) (2025)
- [Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning](https://mlsanity.com/paper/2606.27828/) (2026)
- [Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models](https://mlsanity.com/paper/2509.21984/) (2025)
- [D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition](https://mlsanity.com/paper/2510.08818/) (2025)
- [SafeCoop: Unravelling Full Stack Safety in Agentic Collaborative Driving](https://mlsanity.com/paper/2510.18123/) (2025)
- [Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users](https://mlsanity.com/paper/2602.16930/) (2026)
- [InfoCausalQA:Can Models Perform Non-explicit Causal Reasoning Based on Infographic?](https://mlsanity.com/paper/2508.06220/) (2025)
- [Preliminary Use of Vision Language Model Driven Extraction of Mouse Behavior Towards Understanding Fear Expression](https://mlsanity.com/paper/2510.19160/) (2025)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [To See or To Read: User Behavior Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2511.03845/) (2025)
- [Transforming Behavioral Neuroscience Discovery with In-Context Learning and AI-Enhanced Tensor Methods](https://mlsanity.com/paper/2602.17027/) (2026)
- [On the Cultural Anachronism and Temporal Reasoning in Vision Language Models](https://mlsanity.com/paper/2605.15071/) (2026)
- [Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models](https://mlsanity.com/paper/2606.05702/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved significant advancements in tasks like Visual Question Answering (VQA) by leveraging foundational Large Language Models (LLMs). However, their abilities in specific areas such as visual temporal understanding, which is crucial for comprehending real-world dynamics, remain underexplored. To address this, we propose a challenging evaluation benchmark named TemporalVQA, consisting of two parts: 1) Temporal Order Understanding and 2) Time-lapse Estimation. The first part requires MLLMs to determine the sequence of events by analyzing temporally consecutive video frames. The second part presents image pairs with varying time differences, framed as multiple-choice questions, asking MLLMs to estimate the time-lapse between images with options ranging from seconds to years. Our evaluations of advanced MLLMs, including models like GPT-4o and Gemini-1.5-Pro, reveal significant challenges: GPT-4o achieved only 49.1% average consistent accuracy in temporal order task and 70% in time-lapse estimation, with open-source models performing even poorly. These findings underscore the limitations of current MLLMs in visual temporal understanding and reasoning, highlighting the need for further improvements for their temporal capability. Our dataset can be found at https://huggingface.co/datasets/fazliimam/temporal-vqa.