# OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?

arXiv: https://arxiv.org/abs/2501.05510 · HTML: https://mlsanity.com/paper/2501.05510/

## Key facts

- Cited by 142 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 64 references that are in the map.
- First posted on arXiv 9 Jan 2025.

## Builds on (most cited first, 30 of 48)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering](https://mlsanity.com/paper/1704.04497/) (2017)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding](https://mlsanity.com/paper/2410.17434/) (2024)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Perception Test: A Diagnostic Benchmark for Multimodal Video Models](https://mlsanity.com/paper/2305.13786/) (2023)
- [InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://mlsanity.com/paper/2309.15112/) (2023)

## Built on by (most cited first, 30 of 142)

- [TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos](https://mlsanity.com/paper/2504.17343/) (2025)
- [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://mlsanity.com/paper/2508.09736/) (2025)
- [StreamingVLM: Real-Time Understanding for Infinite Video Streams](https://mlsanity.com/paper/2510.09608/) (2025)
- [StreamForest: Efficient Online Video Understanding with Persistent Event Memory](https://mlsanity.com/paper/2509.24871/) (2025)
- [StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant](https://mlsanity.com/paper/2505.05467/) (2025)
- [OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts](https://mlsanity.com/paper/2503.22952/) (2025)
- [Seed1.8 Model Card: Towards Generalized Real-World Agency](https://mlsanity.com/paper/2603.20633/) (2026)
- [ViSpeak: Visual Instruction Feedback in Streaming Videos](https://mlsanity.com/paper/2503.12769/) (2025)
- [HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding](https://mlsanity.com/paper/2601.14724/) (2026)
- [MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence](https://mlsanity.com/paper/2512.10863/) (2025)
- [Accelerating Streaming Video Large Language Models via Hierarchical Token Compression](https://mlsanity.com/paper/2512.00891/) (2025)
- [StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding](https://mlsanity.com/paper/2508.01875/) (2025)
- [A Simple Baseline for Streaming Video Understanding](https://mlsanity.com/paper/2604.02317/) (2026)
- [FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding](https://mlsanity.com/paper/2603.02096/) (2026)
- [ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Models](https://mlsanity.com/paper/2507.09313/) (2025)
- [Streaming Video Instruction Tuning](https://mlsanity.com/paper/2512.21334/) (2025)
- [LiveStar: Live Streaming Assistant for Real-World Online Video Understanding](https://mlsanity.com/paper/2511.05299/) (2025)
- [Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation](https://mlsanity.com/paper/2510.24821/) (2025)
- [Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously](https://mlsanity.com/paper/2603.12262/) (2026)
- [ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos](https://mlsanity.com/paper/2503.12542/) (2025)
- [RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video](https://mlsanity.com/paper/2505.02064/) (2025)
- [StreamReady: Learning What to Answer and When in Long Streaming Videos](https://mlsanity.com/paper/2603.08620/) (2026)
- [Thinking in Streaming Video](https://mlsanity.com/paper/2603.12938/) (2026)
- [AURA: Always-On Understanding and Real-Time Assistance via Video Streams](https://mlsanity.com/paper/2604.04184/) (2026)
- [RIVER: A Real-Time Interaction Benchmark for Video LLMs](https://mlsanity.com/paper/2603.03985/) (2026)
- [StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios](https://mlsanity.com/paper/2512.04451/) (2025)
- [EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use](https://mlsanity.com/paper/2602.15329/) (2026)
- [Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models](https://mlsanity.com/paper/2506.12776/) (2025)
- [TeleEgo: Benchmarking Egocentric AI Assistants in the Wild](https://mlsanity.com/paper/2510.23981/) (2025)
- [PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarios](https://mlsanity.com/paper/2601.22575/) (2026)

## Abstract (arXiv)

Temporal Awareness, the ability to reason dynamically based on the timestamp when a question is raised, is the key distinction between offline and online video LLMs. Unlike offline models, which rely on complete videos for static, post hoc analysis, online models process video streams incrementally and dynamically adapt their responses based on the timestamp at which the question is posed. Despite its significance, temporal awareness has not been adequately evaluated in existing benchmarks. To fill this gap, we present OVO-Bench (Online-VideO-Benchmark), a novel video benchmark that emphasizes the importance of timestamps for advanced online video understanding capability benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and respond to events occurring at specific timestamps under three distinct scenarios: (1) Backward tracing: trace back to past events to answer the question. (2) Real-time understanding: understand and respond to events as they unfold at the current timestamp. (3) Forward active responding: delay the response until sufficient future information becomes available to answer the question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos and approximately human-curated 2,800 fine-grained meta-annotations with precise timestamps. We combine automated generation pipelines with human curation. With these high-quality samples, we further developed an evaluation pipeline to systematically query video LLMs along the video timeline. Evaluations of nine Video-LLMs reveal that, despite advancements on traditional benchmarks, current models struggle with online video understanding, showing a significant gap compared to human agents. We hope OVO-Bench will drive progress in video LLMs and inspire future research in online video reasoning. Our benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.