mlsanityData to 7 Oct 2026

Seeing the Arrow of Time in Large Multimodal Models

arXiv 2506.03340 ↗First posted 3 Jun 2025cs.CVcited by 28

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

The Arrow of Time (AoT)-time's irreversible flow shaping physical events-is fundamental to video comprehension, yet remains a significant challenge for modern large multimodal models (LMMs). Current LMMs struggle to perceive and utilize temporal directionality in video when responding to language queries, obstructing deeper temporal understanding. We tackle this deficiency by first providing a critical analysis of existing benchmarks and models. We then introduce ArrowRL, a reinforcement learning (RL)-based training strategy with an innovative reverse reward that instills AoT awareness by encouraging divergent video interpretations between forward and reversed visual frames. For rigorous evaluation, we additionally develop AoTBench, a new multi-faceted benchmark probing temporally challenging questions. Experiments show ArrowRL greatly advances temporal perception: it not only achieves substantial improvements on our challenging AoTBench but also demonstrably boosts performance on standard video question answering (VQA) benchmarks (with peak accuracy gains reaching over 20% and 10% respectively). This validates ArrowRL's effectiveness and highlights the critical need for dedicated AoT understanding in LMMs.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 77 of its 90 references that are in the map. First posted on arXiv 3 Jun 2025.

Builds on

77 of 90 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding17 Apr
  2. 2025Video-R1: Reinforcing Video Reasoning in MLLMs27 Mar
  3. 2025Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding28 Jan
  4. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Models13 Dec
  5. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos3 Oct
  6. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  7. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output3 Jul
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  9. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov

Other references · 67

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2025Qwen2.5-VL Technical Reportcited by 5,418
38 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2018World Modelscited by 1,444
  8. 2024A Survey on LLM-as-a-Judgecited by 1,384
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  11. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  13. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  14. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  15. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  16. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  17. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  18. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  19. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  20. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  21. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  22. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 296
  23. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  24. 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
  25. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  26. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  27. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  28. 2019Temporal Cycle-Consistency Learningcited by 151
  29. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  30. 2023Structured World Models from Human Videoscited by 144
  31. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  32. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  33. 2024Improve Vision Language Model Chain-of-thought Reasoningcited by 135
  34. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  35. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  36. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  37. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  38. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 96

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 21.

28 papers.

Sort:

By area:

  1. 2025When Thinking Drifts: Evidential Grounding for Robust Video Reasoningcited by 15
  2. 2026TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMscited by 11
  3. 2025Chirality in Action: Time-Aware Video Representation Learning by Latent Straighteningcited by 10
  4. 2025SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decodingcited by 8
  5. 2026Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Modelscited by 4
  6. 2026YoCausal: How Far is Video Generation from World Model? A Causality Perspectivecited by 3
  7. 2026Can Vision-Language Models Solve the Shell Game?cited by 2
  8. 2026Reinforcing Consistency in Video MLLMs with Structured Rewardscited by 2
  9. 2026STRIVE: Structured Spatiotemporal Exploration for Reinforcement Learning in Video Question Answeringcited by 2
  10. 2026Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMscited by 2
  11. 2025PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding?cited by 1
  12. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
16 more
  1. 2026SHOE: Semantic HOI Open-Vocabulary Evaluation Metriccited by 1
  2. 2026Seeing Fast and Slow: Learning the Flow of Time in Videoscited by 1
  3. 2026Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMscited by 1
  4. 2026Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learningcited by 1
  5. 2026Incentivizing Vision Language Models to Search for Long Video Question Answeringcited by 1
  6. 2026Post-Training VLMs for Video Mistake Detectioncited by 1
  7. 2025Multiscale Video Transformers for Class Agnostic Segmentation in Autonomous Drivingcited by 0
  8. 2025Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videoscited by 0
  9. 2025Adapting MLLMs for Nuanced Video Retrievalcited by 0
  10. 2026OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reorderingcited by 0
  11. 2026The TIME Machine: On The Power of Motion for Efficient Perceptioncited by 0
  12. 2026Diagnosing Under-Development of Irreversible Processes in Video Generationcited by 0
  13. 2026Distributed Implicit Harm: A Compositional Safety Blind Spot in MLLM-Based Video Moderationcited by 0
  14. 2026Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Modelscited by 0
  15. 2026PhyProbe: Rethinking Physical Consistency Evaluation in Generated Videoscited by 0
  16. 2026Before It Fades: Reinforcing Temporal Representations at Inference Time in VideoLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision-Language Models
  2. 2026ArrowGEV: Grounding Events in Video via Learning the Arrow of Time
  3. 2025RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models
  4. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  5. 2025MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding
  6. 2025Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Times
  7. 2025TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
  8. 2025Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!