Watch, Remember, Reason: Human-View Video Understanding with MLLMs
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 202 of its 355 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
202 of 355 references.
Closest ancestors · 44
Same direction, newest first
- 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation3 Feb
- 2026Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning30 Jan
- 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning14 Jan
- 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoning26 Dec
- 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs16 Dec
- 2025Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models28 Nov
- 2025Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning26 Nov
- 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling25 Nov
- 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning17 Nov
- 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning27 Oct
- 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence23 Oct
- 2025Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence23 Oct
32 more
- 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding15 Oct
- 2025ExpVid: A Benchmark for Experiment Video Understanding & Reasoning13 Oct
- 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models6 Oct
- 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning29 Sep
- 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting29 Sep
- 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis28 Sep
- 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception25 Sep
- 2025TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding11 Aug
- 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning6 Aug
- 2025Scaling RL to Long Videos10 Jul
- 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos12 Jun
- 2025CyberV: Cybernetics for Test-time Scaling in Video Understanding9 Jun
- 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos4 Jun
- 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking2 Jun
- 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning31 May
- 2025Reinforcing Video Reasoning with Focused Thinking30 May
- 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?29 May
- 2025VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization25 May
- 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning21 May
- 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning18 May
- 2025VISTA: Mitigating Semantic Inertia in Video-LLMs via Training-Free Dynamic Chain-of-Thought Routing17 May
- 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models8 May
- 2025MINERVA: Evaluating Complex Video Reasoning1 May
- 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning10 Apr
- 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R131 Mar
- 2025Video-R1: Reinforcing Video Reasoning in MLLMs27 Mar
- 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning17 Mar
- 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning14 Mar
- 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models30 Oct
- 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning26 Oct
- 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool7 Jul
- 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition7 May
Other references · 158
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2024GPT-4o System Cardcited by 4,588
- 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
- 2017Dense-Captioning Events in Videoscited by 793
- 2025Qwen2.5-Omni Technical Reportcited by 736
- 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2025Qwen3-Omni Technical Reportcited by 497
- 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
- 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
38 more
- 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
- 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
- 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
- 2025Memory in the Age of AI Agentscited by 303
- 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
- 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 236
- 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
- 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
- 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
- 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
- 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 159
- 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
- 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
- 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
- 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
- 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
- 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
- 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
- 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
- 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
- 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 109
- 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
- 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 95
- 2025PerceptionLM: Open-Access Data and Models for Detailed Visual Understandingcited by 92
- 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
- 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
- 2025Visual Spatial Tuningcited by 89
- 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
- 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
- 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
- 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 80
- 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
- 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
- 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 77
- 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
- 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
- 2025Detect Anything via Next Point Predictioncited by 73
- 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 70
50 most cited of 158 All 158 (JSON)
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understandingcited by 1
- 2026LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agentscited by 0
- 2026Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agentscited by 0
- 2026Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidencecited by 0
- 2026Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence13 Oct 2026
- 2026Thinking Beyond Videos: Unifying Video Reasoning and Deep Research for Open-World Video Agents24 Aug 2026
- 2026StreamFlow: Dynamic Memory Flows for Streaming Video Understanding11 Aug 2026
- 2026LAVE: Latent Visual Evidence-Enhanced Planning for Video Tool-use Agents5 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026WAT: Online Video Understanding Needs Watching Before Thinking
- 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
- 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
- 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
- 2026Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs
- 2026Video-Oasis: Rethinking Evaluation of Video Understanding
- 2025Infinite Video Understanding
- 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models