mlsanityData to 14 Oct 2026

Watch, Remember, Reason: Human-View Video Understanding with MLLMs

arXiv 2606.07433 ↗First posted 5 Jun 2026cs.CV cs.AI cs.MMcited by 4

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Video understanding is being rapidly transformed by multimodal large language models (MLLMs), as research moves from short clips to long, multimodal, and knowledge-intensive video scenarios. These scenarios require models to handle sparse evidence, long-range dependencies, multimodal alignment, and reliable inference under limited computational budgets. This work presents a human-view perspective on LLM-based video understanding, organized around three functional abilities: watching, remembering, and reasoning. Rather than treating video tasks as isolated benchmarks, this view provides a unified structure for analyzing how video MLLMs acquire evidence, preserve context, and produce grounded outputs. We introduce a formulation that characterizes video understanding systems by their perceptual representations, memory states, reasoning traces, and final predictions. Based on this formulation, we identify challenges in spatio-temporal perception, efficient long-video processing, memory modeling, streaming understanding, and faithful reasoning. Representative methods are organized by their roles in video MLLM systems. Watching covers fine-grained, comprehensive, audio-visual, and efficient perception. Remembering includes offline and streaming memory, while reasoning covers text-only reasoning and thinking with videos. We further examine application domains such as egocentric, sports, instructional, medical, and narrative videos, and cover training datasets and evaluation benchmarks across task types, supervision formats, modalities, and capability dimensions. Finally, we outline open problems and future directions for scalable, memory-aware, and evidence-grounded video intelligence. Related works will be continuously traced at https://github.com/marinero4972/Awesome-HumanView-VideoUnderstanding.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 202 of its 355 references that are in the map. First posted on arXiv 5 Jun 2026.

Builds on

202 of 355 references.

Closest ancestors · 44

Same direction, newest first

  1. 2026Video-OPD: Efficient Post-Training of Multimodal Large Language Models for Temporal Video Grounding via On-Policy Distillation3 Feb
  2. 2026Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning30 Jan
  3. 2026Omni-R1: Towards the Unified Generative Paradigm for Multimodal Reasoning14 Jan
  4. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoning26 Dec
  5. 2025TimeLens: Rethinking Video Temporal Grounding with Multimodal LLMs16 Dec
  6. 2025Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models28 Nov
  7. 2025Thinking With Bounding Boxes: Enhancing Spatio-Temporal Video Grounding via Reinforcement Fine-Tuning26 Nov
  8. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Calling25 Nov
  9. 2025DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning17 Nov
  10. 2025Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning27 Oct
  11. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence23 Oct
  12. 2025Conan: Progressive Learning to Reason Like a Detective over Multi-Scale Visual Evidence23 Oct
32 more
  1. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding15 Oct
  2. 2025ExpVid: A Benchmark for Experiment Video Understanding & Reasoning13 Oct
  3. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models6 Oct
  4. 2025LOVE-R1: Advancing Long Video Understanding with an Adaptive Zoom-in Mechanism via Multi-Step Reasoning29 Sep
  5. 2025FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting29 Sep
  6. 2025ReWatch-R1: Boosting Complex Video Reasoning in Large Vision-Language Models through Agentic Data Synthesis28 Sep
  7. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception25 Sep
  8. 2025TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding11 Aug
  9. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning6 Aug
  10. 2025Scaling RL to Long Videos10 Jul
  11. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos12 Jun
  12. 2025CyberV: Cybernetics for Test-time Scaling in Video Understanding9 Jun
  13. 2025MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos4 Jun
  14. 2025VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking2 Jun
  15. 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning31 May
  16. 2025Reinforcing Video Reasoning with Focused Thinking30 May
  17. 2025VideoReasonBench: Can MLLMs Perform Vision-Centric Complex Video Reasoning?29 May
  18. 2025VerIPO: Cultivating Long Reasoning in Video-LLMs via Verifier-Gudied Iterative Policy Optimization25 May
  19. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning21 May
  20. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning18 May
  21. 2025VISTA: Mitigating Semantic Inertia in Video-LLMs via Training-Free Dynamic Chain-of-Thought Routing17 May
  22. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models8 May
  23. 2025MINERVA: Evaluating Complex Video Reasoning1 May
  24. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoning10 Apr
  25. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R131 Mar
  26. 2025Video-R1: Reinforcing Video Reasoning in MLLMs27 Mar
  27. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoning17 Mar
  28. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning14 Mar
  29. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models30 Oct
  30. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoning26 Oct
  31. 2024VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool7 Jul
  32. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition7 May

Other references · 158

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  4. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2017Dense-Captioning Events in Videoscited by 793
  7. 2025Qwen2.5-Omni Technical Reportcited by 736
  8. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  9. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  10. 2025Qwen3-Omni Technical Reportcited by 497
  11. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  12. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
38 more
  1. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  2. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  3. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  4. 2025Memory in the Age of AI Agentscited by 303
  5. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  6. 2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 236
  7. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  8. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 173
  9. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  10. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  11. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 159
  12. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  13. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  14. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  15. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  16. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  17. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  18. 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
  19. 2025Cambrian-S: Towards Spatial Supersensing in Videocited by 134
  20. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  21. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 109
  22. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  23. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 95
  24. 2025PerceptionLM: Open-Access Data and Models for Detailed Visual Understandingcited by 92
  25. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  26. 2025SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Modelscited by 90
  27. 2025Visual Spatial Tuningcited by 89
  28. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
  29. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  30. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  31. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 80
  32. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 79
  33. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  34. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 77
  35. 2025LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive Streaming Speech Synthesiscited by 76
  36. 2025Kwai Keye-VL 1.5 Technical Reportcited by 73
  37. 2025Detect Anything via Next Point Predictioncited by 73
  38. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 70

50 most cited of 158 All 158 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026WAT: Online Video Understanding Needs Watching Before Thinking
  2. 2026Moment-Video: Diagnosing Temporal Fidelity of Video MLLMs on Momentary Visual Events
  3. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
  4. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
  5. 2026Beyond Frame Selection: Rethinking Long-Video Understanding with MLLMs
  6. 2026Video-Oasis: Rethinking Evaluation of Video Understanding
  7. 2025Infinite Video Understanding
  8. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models