mlsanityData to 14 Oct 2026

VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos

arXiv 2405.19209 ↗First posted 29 May 2024cs.CV cs.AI cs.CLcited by 251

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Long-form video understanding is complicated by the high redundancy of video data and the abundance of query-irrelevant information. To tackle these challenges, we propose VideoTree, a training-free framework which builds a query-adaptive and hierarchical video representation for LLM reasoning over long-form videos. First, VideoTree extracts query-relevant information from the input video through an iterative process, progressively refining the selection of keyframes based on their relevance to the query. Furthermore, VideoTree leverages the inherent hierarchical structure of long video data, which is often overlooked by existing LLM-based methods. Specifically, we incorporate multi-granularity information into a tree-based representation, allowing VideoTree to extract query-relevant details from long videos in a coarse-to-fine manner. This enables the model to effectively handle a wide range of video queries with varying levels of detail. Finally, VideoTree aggregates the hierarchical query-relevant information within the tree structure and feeds it into an LLM reasoning model to answer the query. Our experiments show that our method improves both reasoning accuracy and efficiency. Specifically, VideoTree outperforms existing training-free approaches on EgoSchema and NExT-QA with less inference time, achieving 61.1% and 75.6% accuracy on the test set without additional video-specific training. Moreover, on the long split of Video-MME (average 44 minutes), VideoTree achieves better performance than GPT-4V and many other MLLMs that were extensively trained on video data.

Key facts. Cited by 251 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 67 of its 107 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

67 of 107 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SOK-Bench: A Situated Video Reasoning Benchmark with Aligned Open-World Knowledge15 May
  2. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answering9 Apr
  3. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM27 Mar
  4. 2024TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning29 Feb
  5. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answering24 Oct
  6. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar

Other references · 61

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  11. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  12. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
38 more
  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  2. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  3. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  4. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  5. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  6. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  7. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  8. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  9. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  10. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  11. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  12. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  13. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  14. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  15. 2022Learning Video Representations from Large Language Modelscited by 205
  16. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  17. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  18. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  19. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  20. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  21. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  22. 2022Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learningcited by 90
  23. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 89
  24. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  25. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  26. 2022TALLFormer: Temporal Action Localization with a Long-memory Transformercited by 79
  27. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  28. 2019LiteEval: A Coarse-to-Fine Framework for Resource Efficient Video Recognitioncited by 65
  29. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  30. 2022Video Graph Transformer for Video Question Answeringcited by 63
  31. 2024DoraemonGPT: Toward Understanding Dynamic Scenes with Large Language Models (Exemplified as A Video Agent)cited by 58
  32. 2024Language Repository for Long Video Understandingcited by 55
  33. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 51
  34. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  35. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  36. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  37. 2021Long-Short Temporal Contrastive Learning of Video Transformerscited by 42
  38. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 23; 2025: 106; 2026: 122.

251 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  2. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 167
  3. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  4. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  5. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  6. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
  7. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 98
  8. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  9. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 83
  10. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  11. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  12. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
38 more
  1. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  2. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 65
  3. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 65
  4. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65
  5. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  6. 2024VCA: Video Curious Agent for Long Video Understandingcited by 56
  7. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  8. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 49
  9. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  10. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 41
  11. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 38
  12. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 37
  13. 2024AMEGO: Active Memory from long EGOcentric videoscited by 36
  14. 2024OmAgent: A Multi-modal Agent Framework for Complex Video Understanding with Task Divide-and-Conquercited by 34
  15. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  16. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 34
  17. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 33
  18. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  19. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  20. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 31
  21. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  22. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  23. 2025VideoExplorer: Think With Videos For Agentic Long-Video Understandingcited by 29
  24. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 29
  25. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
  26. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  27. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Explorationcited by 25
  28. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  29. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 24
  30. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24
  31. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  32. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  33. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 22
  34. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  35. 2025TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learningcited by 20
  36. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 19
  37. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  38. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19

50 most cited of 251 All 251 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning
  2. 2023Long Video Understanding with Learnable Retrieval in Video-Language Models
  3. 2025SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding
  4. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  5. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?
  6. 2026See More, Think Deeper: Query-Expanded Visual Evidence and Answer-Clue Guided Reflection for Long Video Understanding
  7. 2023VideoLLM: Modeling Video Sequence with Large Language Models
  8. 2026Reasoning with Memory: A Temporal Granularity-Adaptive Framework for Training-Free Long Video Understanding