Video Understanding: From Geometry and Semantics to Unified Models
Not on the map. Why?
Abstract (arXiv)
Video understanding aims to enable models to perceive, reason about, and interact with the dynamic visual world. In contrast to image understanding, video understanding inherently requires modeling temporal dynamics and evolving visual context, placing stronger demands on spatiotemporal reasoning and making it a foundational problem in computer vision. In this survey, we present a structured overview of video understanding by organizing the literature into three complementary perspectives: low-level video geometry understanding, high-level semantic understanding, and unified video understanding models. We further highlight a broader shift from isolated, task-specific pipelines toward unified modeling paradigms that can be adapted to diverse downstream objectives, enabling a more systematic view of recent progress. By consolidating these perspectives, this survey provides a coherent map of the evolving video understanding landscape, summarizes key modeling trends and design principles, and outlines open challenges toward building robust, scalable, and unified video foundation models.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 352 of its 404 references that are in the map. First posted on arXiv 18 Mar 2026.
Builds on
352 of 404 references.
References · 352
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Visual Instruction Tuningcited by 7,792
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
38 more
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
- 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
- 2022Video Diffusion Modelscited by 2,280
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
- 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,437
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
- 2024Depth Anything V2cited by 1,385
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
- 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
- 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
- 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
- 2024Grounding Image Matching in 3D with MASt3Rcited by 960
- 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2025SAM 3: Segment Anything with Conceptscited by 835
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2023VILA: On Pre-training for Visual Language Modelscited by 686
- 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
- 2017Localizing Moments in Video with Natural Languagecited by 660
- 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
- 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
- 2023LightGlue: Local Feature Matching at Light Speedcited by 583
- 2017TALL: Temporal Activity Localization via Language Querycited by 581
- 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
50 most cited of 352 All 352 (JSON)
Built on by
Citing papers per year: 2026: 5.
5 papers.
- 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 19
- 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generationcited by 17
- 2026Stitched Value Model for Diffusion Alignmentcited by 2
- 2026SubjectAnchor: Subject-Aware Memory-to-Video for Multi-Shot Storytellingcited by 1
- 2026Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Trackscited by 0
- 2026SubjectAnchor: Subject-Aware Memory-to-Video for Multi-Shot Storytelling28 Sep 2026
- 2026Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks14 Jun 2026
- 2026Stitched Value Model for Diffusion Alignment19 May 2026
- 2026Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation27 Apr 2026
- 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Reward27 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026Video Understanding: Through A Temporal Lens
- 2026Video-Oasis: Rethinking Evaluation of Video Understanding
- 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Models
- 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding
- 2025Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought
- 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition
- 2026VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocation
- 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models