mlsanityData to 14 Oct 2026

Burn After Reading: Do Multimodal Large Language Models Truly Capture Order of Events in Image Sequences?

arXiv 2506.10415 ↗First posted 12 Jun 2025cs.CL cs.CVcited by 4

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

This paper introduces the TempVS benchmark, which focuses on temporal grounding and reasoning capabilities of Multimodal Large Language Models (MLLMs) in image sequences. TempVS consists of three main tests (i.e., event relation inference, sentence ordering and image ordering), each accompanied with a basic grounding test. TempVS requires MLLMs to rely on both visual and linguistic modalities to understand the temporal order of events. We evaluate 38 state-of-the-art MLLMs, demonstrating that models struggle to solve TempVS, with a substantial performance gap compared to human capabilities. We also provide fine-grained insights that suggest promising directions for future research. Our TempVS benchmark data and code are available at https://github.com/yjsong22/TempVS.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 43 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

29 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  2. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 817
15 more
  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  2. 2024Long Context Transfer from Language to Visioncited by 515
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  4. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  5. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  6. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  7. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  8. 2016Visual Storytellingcited by 176
  9. 2018StoryGAN: A Sequential Conditional GAN for Story Visualizationcited by 138
  10. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  11. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  12. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 39
  13. 2023Visual Writing Prompts: Character-Grounded Story Generation with Curated Image Sequencescited by 31
  14. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  15. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026TPRU: Advancing Temporal and Procedural Understanding in Large Multimodal Modelscited by 2
  2. 2025Can we use LLMs to bootstrap reinforcement learning? -- A case study in digital health behavior changecited by 0
  3. 2026What's the Catch? Evaluating Temporal Consistency in Vision-Language Modelscited by 0
  4. 2026From Vision to Language: Investigating Causal Information Flow in Multimodal Decision-Makingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models
  2. 2026Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequences
  3. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Models
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  5. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!
  6. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  7. 2024The Revolution of Multimodal Large Language Models: A Survey
  8. 2026Same Evidence, Different Answer: Auditing Order Sensitivity in Multimodal Large Language Models