# StreamForest: Efficient Online Video Understanding with Persistent Event Memory

arXiv: https://arxiv.org/abs/2509.24871 · HTML: https://mlsanity.com/paper/2509.24871/

## Key facts

- Cited by 78 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 73 of its 82 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 73)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [TALL: Temporal Activity Localization via Language Query](https://mlsanity.com/paper/1705.02101/) (2017)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [YOLOv11: An Overview of the Key Architectural Enhancements](https://mlsanity.com/paper/2410.17725/) (2024)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)

## Built on by (most cited first, 30 of 78)

- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026)
- [Accelerating Streaming Video Large Language Models via Hierarchical Token Compression](https://mlsanity.com/paper/2512.00891/) (2025)
- [A Simple Baseline for Streaming Video Understanding](https://mlsanity.com/paper/2604.02317/) (2026)
- [FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding](https://mlsanity.com/paper/2603.02096/) (2026)
- [Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously](https://mlsanity.com/paper/2603.12262/) (2026)
- [StreamReady: Learning What to Answer and When in Long Streaming Videos](https://mlsanity.com/paper/2603.08620/) (2026)
- [Thinking in Streaming Video](https://mlsanity.com/paper/2603.12938/) (2026)
- [RIVER: A Real-Time Interaction Benchmark for Video LLMs](https://mlsanity.com/paper/2603.03985/) (2026)
- [JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence](https://mlsanity.com/paper/2606.14777/) (2026)
- [EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use](https://mlsanity.com/paper/2602.15329/) (2026)
- [TeleEgo: Benchmarking Egocentric AI Assistants in the Wild](https://mlsanity.com/paper/2510.23981/) (2025)
- [VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding](https://mlsanity.com/paper/2607.14935/) (2026)
- [Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model](https://mlsanity.com/paper/2607.24904/) (2026)
- [PEARL: Personalized Streaming Video Understanding Model](https://mlsanity.com/paper/2603.20422/) (2026)
- [StreamMeCo: Long-Term Agent Memory Compression for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2604.09000/) (2026)
- [Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2603.11896/) (2026)
- [Semantic-Aware Adaptive Visual Memory for Streaming Video Understanding](https://mlsanity.com/paper/2605.07897/) (2026)
- [Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding](https://mlsanity.com/paper/2603.19054/) (2026)
- [What Should a Streaming Video Model Remember?](https://mlsanity.com/paper/2606.16353/) (2026)
- [OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMs](https://mlsanity.com/paper/2606.03890/) (2026)
- [CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management](https://mlsanity.com/paper/2603.19571/) (2026)
- [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](https://mlsanity.com/paper/2604.06036/) (2026)
- [StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering](https://mlsanity.com/paper/2605.25621/) (2026)
- [ObjectStream: Latent Objects as Memory Anchors for Streaming Video Understanding](https://mlsanity.com/paper/2607.28312/) (2026)
- [StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs](https://mlsanity.com/paper/2608.13416/) (2026)
- [video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLM](https://mlsanity.com/paper/2510.11129/) (2025)
- [Don't Pause! Every prediction matters in a streaming video](https://mlsanity.com/paper/2604.24317/) (2026)
- [StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video](https://mlsanity.com/paper/2605.16381/) (2026)
- [OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants](https://mlsanity.com/paper/2605.26485/) (2026)
- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have recently achieved remarkable progress in video understanding. However, their effectiveness in real-time streaming scenarios remains limited due to storage constraints of historical visual features and insufficient real-time spatiotemporal reasoning. To address these challenges, we propose StreamForest, a novel architecture specifically designed for streaming video understanding. Central to StreamForest is the Persistent Event Memory Forest, a memory mechanism that adaptively organizes video frames into multiple event-level tree structures. This process is guided by penalty functions based on temporal distance, content similarity, and merge frequency, enabling efficient long-term memory retention under limited computational resources. To enhance real-time perception, we introduce a Fine-grained Spatiotemporal Window, which captures detailed short-term visual cues to improve current scene perception. Additionally, we present OnlineIT, an instruction-tuning dataset tailored for streaming video tasks. OnlineIT significantly boosts MLLM performance in both real-time perception and future prediction. To evaluate generalization in practical applications, we introduce ODV-Bench, a new benchmark focused on real-time streaming video understanding in autonomous driving scenarios. Experimental results demonstrate that StreamForest achieves the state-of-the-art performance, with accuracies of 77.3% on StreamingBench, 60.5% on OVBench, and 55.6% on OVO-Bench. In particular, even under extreme visual token compression (limited to 1024 tokens), the model retains 96.8% of its average accuracy in eight benchmarks relative to the default setting. These results underscore the robustness, efficiency, and generalizability of StreamForest for streaming video understanding.