# VRR-QA: Visual Relational Reasoning in Videos Beyond Explicit Cues

arXiv: https://arxiv.org/abs/2506.21742 · HTML: https://mlsanity.com/paper/2506.21742/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 38 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Builds on (most cited first, 22 of 22)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [TempCompass: Do Video LLMs Really Understand Videos?](https://mlsanity.com/paper/2403.00476/) (2024)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)
- [Evaluating the Evaluation of Diversity in Natural Language Generation](https://mlsanity.com/paper/2004.02990/) (2020)
- [VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding](https://mlsanity.com/paper/2406.09418/) (2024)
- [CinePile: A Long Video Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2405.08813/) (2024)
- [Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding](https://mlsanity.com/paper/2501.07888/) (2025)
- [ViLMA: A Zero-Shot Benchmark for Linguistic and Temporal Grounding in Video-Language Models](https://mlsanity.com/paper/2311.07022/) (2023)
- [Commonsense Video Question Answering through Video-Grounded Entailment Tree Reasoning](https://mlsanity.com/paper/2501.05069/) (2025)
- [TimeLogic: A Temporal Logic Benchmark for Video QA](https://mlsanity.com/paper/2501.07214/) (2025)
- [ReasVQA: Advancing VideoQA with Imperfect Reasoning Process](https://mlsanity.com/paper/2501.13536/) (2025)

## Built on by (most cited first, 8 of 8)

- [The Telephone Game: Evaluating Semantic Drift in Unified Models](https://mlsanity.com/paper/2509.04438/) (2025)
- [Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos](https://mlsanity.com/paper/2510.11204/) (2025)
- [Text-Driven Reasoning Video Editing via Reinforcement Learning on Digital Twin Representations](https://mlsanity.com/paper/2511.14100/) (2025)
- [Answer Self-Consistency with Margin-Triggered Question Re-Arbitration for the CVPR 2026 VidLLMs Challenge](https://mlsanity.com/paper/2606.04323/) (2026)
- [Adaptive Dense Evidence Refinement for Video Relational Reasoning for VRR-QA Challenge](https://mlsanity.com/paper/2606.01104/) (2026)
- [Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering](https://mlsanity.com/paper/2606.01485/) (2026)
- [Question-Aware Evidence Ledgers for Video Relational Reasoning](https://mlsanity.com/paper/2606.02506/) (2026)
- [DocMIDE: Learning Multi-Hop Implicit Derivation in Visually Rich Documents](https://mlsanity.com/paper/2609.24092/) (2026)

## Abstract (arXiv)

Video Question Answering (VideoQA) has made significant strides by leveraging multimodal learning to align visual and textual modalities. However, current benchmarks overwhelmingly focus on questions answerable through explicit visual content - actions, objects, and events - directly observable within individual frames or short clips. To truly understand videos as humans do, models must go beyond what is directly shown, inferring hidden relationships and contextual cues that are only implied across frames. Current benchmarks fail to capture this essential aspect of video understanding. To address this gap, we introduce VRR-QA, a benchmark for Visual Relational Reasoning Beyond Explicit Cues. We curate our benchmark from creative and cinematic videos such as movies, that deliberately employ storytelling techniques which omit direct depictions of certain events or relations, requiring viewers to infer them. VRR-QA comprises 1K meticulously expert-annotated QA pairs drawn from 1K creative video clips covering 15 genres across 7 decades of content, from both live-action and animated titles. Our extensive evaluations on 14 leading VideoQA models reveals consistent and significant performance degradation, underscoring their reliance on surface-level visual cues and highlighting the difficulty of implicit reasoning. Even the best model substantially underperforms human baselines with only 64% accuracy. Performance variations across models further illustrate the complexity and diversity of the challenges presented by VRR-QA. By releasing both dataset and data collection framework, VRR-QA establishes a rigorous, diverse, and reproducible testbed for advancing VideoQA: https://swetha5.github.io/ImplicitQA/.