# VideoForest: Person-Anchored Hierarchical Reasoning for Cross-Video Question Answering

arXiv: https://arxiv.org/abs/2508.03039 · HTML: https://mlsanity.com/paper/2508.03039/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 30 of 31)

- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling](https://mlsanity.com/paper/2501.00574/) (2024)
- [InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling](https://mlsanity.com/paper/2501.12386/) (2025)
- [Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos](https://mlsanity.com/paper/2408.14023/) (2024)
- [The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Pose](https://mlsanity.com/paper/2007.00394/) (2020)
- [Video ReCap: Recursive Captioning of Hour-Long Videos](https://mlsanity.com/paper/2402.13250/) (2024)
- [TransVOD: End-to-End Video Object Detection with Spatial-Temporal Transformers](https://mlsanity.com/paper/2201.05047/) (2022)
- [VideoVista: A Versatile Benchmark for Video Understanding and Reasoning](https://mlsanity.com/paper/2406.11303/) (2024)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA](https://mlsanity.com/paper/2406.09396/) (2024)
- [ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System](https://mlsanity.com/paper/2304.14407/) (2023)
- [DrVideo: Document Retrieval Based Long Video Understanding](https://mlsanity.com/paper/2406.12846/) (2024)
- [Foundation Models for Video Understanding: A Survey](https://mlsanity.com/paper/2405.03770/) (2024)
- [Enhancing Video-Language Representations with Structural Spatio-Temporal Alignment](https://mlsanity.com/paper/2406.19255/) (2024)
- [Enhancing Long Video Understanding via Hierarchical Event-Based Memory](https://mlsanity.com/paper/2409.06299/) (2024)
- [TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning](https://mlsanity.com/paper/2402.19467/) (2024)

## Abstract (arXiv)

Cross-video question answering presents significant challenges beyond traditional single-video understanding, particularly in establishing meaningful connections across video streams and managing the complexity of multi-source information retrieval. We introduce VideoForest, a novel framework that addresses these challenges through person-anchored hierarchical reasoning. Our approach leverages person-level features as natural bridge points between videos, enabling effective cross-video understanding without requiring end-to-end training. VideoForest integrates three key innovations: 1) a human-anchored feature extraction mechanism that employs ReID and tracking algorithms to establish robust spatiotemporal relationships across multiple video sources; 2) a multi-granularity spanning tree structure that hierarchically organizes visual content around person-level trajectories; and 3) a multi-agent reasoning framework that efficiently traverses this hierarchical structure to answer complex cross-video queries. To evaluate our approach, we develop CrossVideoQA, a comprehensive benchmark dataset specifically designed for person-centric cross-video analysis. Experimental results demonstrate VideoForest's superior performance in cross-video reasoning tasks, achieving 71.93% accuracy in person recognition, 83.75% in behavior analysis, and 51.67% in summarization and reasoning, significantly outperforming existing methods. Our work establishes a new paradigm for cross-video understanding by unifying multiple video streams through person-level features, enabling sophisticated reasoning across distributed visual information while maintaining computational efficiency.