mlsanityData to 7 Oct 2026

ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos

arXiv 2406.19392 ↗First posted 27 Jun 2024cs.CVcited by 51

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We introduce ReXTime, a benchmark designed to rigorously test AI models' ability to perform temporal reasoning within video events. Specifically, ReXTime focuses on reasoning across time, i.e. human-like understanding when the question and its corresponding answer occur in different video segments. This form of reasoning, requiring advanced understanding of cause-and-effect relationships across video segments, poses significant challenges to even the frontier multimodal large language models. To facilitate this evaluation, we develop an automated pipeline for generating temporal reasoning question-answer pairs, significantly reducing the need for labor-intensive manual annotations. Our benchmark includes 921 carefully vetted validation samples and 2,143 test samples, each manually curated for accuracy and relevance. Evaluation results show that while frontier large language models outperform academic models, they still lag behind human performance by a significant 14.3% accuracy gap. Additionally, our pipeline creates a training dataset of 9,695 machine generated samples without manual effort, which empirical studies suggest can enhance the across-time reasoning via fine-tuning.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 65 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

40 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning18 Feb

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
27 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  8. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  9. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  10. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  11. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  12. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  13. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  14. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 234
  15. 2023UniVTG: Towards Unified Video-Language Temporal Groundingcited by 189
  16. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  17. 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
  18. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  19. 2018Multilevel Language and Vision Integration for Text-to-Clip Retrievalcited by 128
  20. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  21. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  22. 2018Localizing Moments in Video with Temporal Languagecited by 83
  23. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 83
  24. 2023Knowing Where to Focus: Event-aware Transformer for Video Groundingcited by 79
  25. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  26. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  27. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 32; 2026: 15.

51 papers.

Sort:

By area:

  1. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  2. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  3. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 82
  4. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  5. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  6. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  7. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  8. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  9. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  10. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 24
  11. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  12. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
38 more
  1. 2025Fostering Video Reasoning via Next-Event Predictioncited by 15
  2. 2024Progress-Aware Video Frame Captioningcited by 10
  3. 2025ZoomV: Temporal Zoom-in for Efficient Long Video Understandingcited by 10
  4. 2025TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervisioncited by 8
  5. 2025AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyondcited by 8
  6. 2025Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-incited by 8
  7. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  8. 2025VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotationscited by 7
  9. 2025KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understandingcited by 6
  10. 2026VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videoscited by 6
  11. 2025MECD+: Unlocking Event-Level Causal Graph Discovery for Video Reasoningcited by 5
  12. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  13. 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
  14. 2025MUPA: Towards Multi-Path Agentic Reasoning for Grounded Video Question Answeringcited by 4
  15. 2025Bidirectional Likelihood Estimation with Multi-Modal Large Language Models for Text-Video Retrievalcited by 4
  16. 2025Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoningcited by 3
  17. 2025NeMo: Needle in a Montage for Video-Language Understandingcited by 3
  18. 2026EvoGround: Self-Evolving Video Agents for Video Temporal Groundingcited by 3
  19. 2025What Happens When: Learning Temporal Orders of Events in Videoscited by 2
  20. 2026VidLaDA: Bidirectional Diffusion Large Language Models for Efficient Video Understandingcited by 2
  21. 2026It's a Matter of Time: Three Lessons on Long-Term Motion for Perceptioncited by 2
  22. 2026Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Modelscited by 2
  23. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  24. 2025VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understandingcited by 1
  25. 2025AgentCaster: Reasoning-Guided Tornado Forecastingcited by 1
  26. 2025ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videoscited by 1
  27. 2026TA-Prompting: Enhancing Video Large Language Models for Dense Video Captioning via Temporal Anchorscited by 1
  28. 2026DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understandingcited by 1
  29. 2026EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidencecited by 1
  30. 2025LogSTOP: Temporal Scores over Prediction Sequences for Matching and Retrievalcited by 0
  31. 2025Understanding Temporal Logic Consistency in Video-Language Models through Cross-Modal Attention Discriminabilitycited by 0
  32. 2025A Multimodal Benchmark for Framing of Oil & Gas Advertising and Potential Greenwashing Detectioncited by 0
  33. 2025Trust in Vision-Language Models: Insights from a Participatory User Workshopcited by 0
  34. 2026RiskCueBench: Benchmarking Anticipatory Reasoning from Early Risk Cues in Video-Language Modelscited by 0
  35. 2026Narrative Aligned Long Form Video Question Answeringcited by 0
  36. 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMscited by 0
  37. 2026X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understandingcited by 0
  38. 2026NEST: Narrative Event Structures in Time for Long Video Understandingcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering
  2. 2025Deep Temporal Reasoning in Video Language Models: A Cross-Linguistic Evaluation of Action Duration and Completion through Perfect Times
  3. 2026TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Living
  4. 2026SYNCR: A Cross-Video Reasoning Benchmark with Synthetic Grounding
  5. 2026TimeLogic Challenge @ CVPR 2026: Strong MLLMs Meet Evidence-Seeking Agents for Temporal-Logic Video Question Answering ↗
  6. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  7. 2026TLG: Temporal-Logic Grounding for Video Question Answering via Source-Annotation Reconstruction and Category-Targeted Reasoning
  8. 2025RTime-QA: A Benchmark for Atomic Temporal Event Understanding in Large Multi-modal Models