mlsanityData to 7 Oct 2026

Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs

arXiv 2502.05092 ↗First posted 7 Feb 2025cs.CV cs.AI cs.CLcited by 10

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Understanding time from visual representations is a fundamental cognitive skill, yet it remains a challenge for multimodal large language models (MLLMs). In this work, we investigate the capabilities of MLLMs in interpreting time and date through analogue clocks and yearly calendars. To facilitate this, we curated a structured dataset comprising two subsets: 1) textitClockQA, which comprises various types of clock styles-standard, black-dial, no-second-hand, Roman numeral, and arrow-hand clocks-paired with time related questions; and 2) textitCalendarQA, which consists of yearly calendar images with questions ranging from commonly known dates (e.g., Christmas, New Year's Day) to computationally derived ones (e.g., the 100th or 153rd day of the year). We aim to analyse how MLLMs can perform visual recognition, numerical reasoning, and temporal inference when presented with time-related visual data. Our evaluations show that despite recent advancements, reliably understanding time remains a significant challenge for MLLMs.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 11 of its 18 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

11 of 18 references.

References · 11

  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  3. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  6. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  7. 2024DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLMcited by 45
  8. 2024ReMI: A Dataset for Reasoning with Multiple Imagescited by 27
  9. 2024Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoningcited by 23
  10. 2021It's About Time: Analog Clock Reading in the Wildcited by 14
  11. 2024Evidence of Cognitive Deficits andDevelopmental Advances in Generative AI: A Clock Drawing Test Analysiscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2026Large Language Model Reasoning Failurescited by 20
  2. 2025Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBenchcited by 6
  3. 2025Have Multimodal Large Language Models (MLLMs) Really Learned to Tell the Time on Analog Clocks?cited by 4
  4. 2026Measuring Iterative Temporal Reasoning with Time Puzzlescited by 1
  5. 2026It's Time to Get It Right: Improving Analog Clock Reading and Clock-Hand Spatial Reasoning in Vision-Language Modelscited by 1
  6. 2026What Really Controls Temporal Reasoning in Large Language Models: Tokenisation or Representation of Time?cited by 1
  7. 2025SPAN: Benchmarking and Improving Cross-Calendar Temporal Reasoning of Large Language Modelscited by 0
  8. 2026The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Spacecited by 0
  9. 2026Blind-Spots-Bench: Evaluating Blind Spots in Multimodal Modelscited by 0
  10. 2026It's Always 10:10: Reference Images Break a Bias That Prompts Only Dentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!
  2. 2024A Picture is Worth A Thousand Numbers: Enabling LLMs Reason about Time Series via Visualization
  3. 2025A Matter of Time: Revealing the Structure of Time in Vision-Language Models
  4. 2024Chrono: A Simple Blueprint for Representing Time in MLLMs
  5. 2025MLLM4TS: Leveraging Vision and Multimodal Language Models for General Time-Series Analysis
  6. 2026Seeing Time: Benchmarking Chronological Reasoning and Shortcut Biases in Vision-Language Models
  7. 2024Do Language Models Understand Time?
  8. 2025TimeSense:Making Large Language Models Proficient in Time-Series Analysis