mlsanityData to 14 Oct 2026

STAR: A Benchmark for Situated Reasoning in Real-World Videos

arXiv 2405.09711 ↗First posted 15 May 2024cs.AI cs.CL cs.CVcited by 213

Not on the map. Why?

Cited by 91 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Reasoning in the real world is not divorced from situations. How to capture the present knowledge from surrounding situations and perform reasoning accordingly is crucial and challenging for machine intelligence. This paper introduces a new benchmark that evaluates the situated reasoning ability via situation abstraction and logic-grounded question answering for real-world videos, called Situated Reasoning in Real-World Videos (STAR Benchmark). This benchmark is built upon the real-world videos associated with human actions or interactions, which are naturally dynamic, compositional, and logical. The dataset includes four types of questions, including interaction, sequence, prediction, and feasibility. We represent the situations in real-world videos by hyper-graphs connecting extracted atomic entities and relations (e.g., actions, persons, objects, and relationships). Besides visual perception, situated reasoning also requires structured situation comprehension and logical reasoning. Questions and answers are procedurally generated. The answering logic of each question is represented by a functional program based on a situation hyper-graph. We compare various existing video reasoning models and find that they all struggle on this challenging situated reasoning task. We further propose a diagnostic neuro-symbolic model that can disentangle visual perception, situation abstraction, language understanding, and functional reasoning to understand the challenges of this benchmark.

Key facts. Cited by 213 papers in the map (data through 14 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 15 May 2024.

Builds on

37 of 50 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  5. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  10. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  12. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
25 more
  1. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  2. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  3. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  4. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  5. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  6. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  7. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  8. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  9. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  10. 2020GECToR -- Grammatical Error Correction: Tag, Not Rewritecited by 177
  11. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  12. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 154
  13. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  14. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  15. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  16. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  17. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  18. 2019Language-Conditioned Graph Networks for Relational Reasoningcited by 55
  19. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  20. 2017VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentationcited by 50
  21. 2016MarioQA: Answering Questions by Watching Gameplay Videoscited by 42
  22. 2021PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoningcited by 34
  23. 2015Actions ~ Transformationscited by 30
  24. 2017What Actions are Needed for Understanding Human Actions in Videos?cited by 29
  25. 2018A Dataset and Architecture for Visual Reasoning with a Working Memorycited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 64; 2025: 91; 2026: 58.

213 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  5. 2024Long Context Transfer from Language to Visioncited by 515
  6. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  7. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  8. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  9. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  10. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  11. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  12. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
38 more
  1. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  2. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  3. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  4. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98
  5. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
  6. 2025Scaling RL to Long Videoscited by 84
  7. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 83
  8. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  9. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  10. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  11. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  12. 2024Online Video Understanding: OVBench and VideoChat-Onlinecited by 61
  13. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
  14. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  15. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
  16. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 45
  17. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  18. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 42
  19. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlightingcited by 42
  20. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  21. 2024Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillationcited by 37
  22. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 37
  23. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 36
  24. 2025UrbanVideo-Bench: Benchmarking Vision-Language Models on Embodied Intelligence with Video Data in Urban Spacescited by 36
  25. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  26. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learningcited by 34
  27. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 33
  28. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 32
  29. 2025Reinforcing Video Reasoning with Focused Thinkingcited by 31
  30. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  31. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  32. 2024Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervisioncited by 25
  33. 2024Neptune: The Long Orbit to Benchmarking Long Video Understandingcited by 25
  34. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  35. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 24
  36. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  37. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  38. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 24

50 most cited of 213 All 213 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge
  2. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
  3. 2026PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning
  4. 2026CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering
  5. 2024BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind
  6. 2025CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models
  7. 2026Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning
  8. 2026AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents