# LiveStar: Live Streaming Assistant for Real-World Online Video Understanding

arXiv: https://arxiv.org/abs/2511.05299 · HTML: https://mlsanity.com/paper/2511.05299/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 66 of its 77 references that are in the map.
- First posted on arXiv 7 Nov 2025.

## Builds on (most cited first, 30 of 66)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering](https://mlsanity.com/paper/1906.02467/) (2019)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [TVQA: Localized, Compositional Video Question Answering](https://mlsanity.com/paper/1809.01696/) (2018)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding](https://mlsanity.com/paper/2312.02051/) (2023)

## Built on by (most cited first, 25 of 25)

- [A Simple Baseline for Streaming Video Understanding](https://mlsanity.com/paper/2604.02317/) (2026)
- [Proact-VL: A Proactive VideoLLM for Real-Time AI Companions](https://mlsanity.com/paper/2603.03447/) (2026)
- [Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2603.11896/) (2026)
- [What Should a Streaming Video Model Remember?](https://mlsanity.com/paper/2606.16353/) (2026)
- [ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding](https://mlsanity.com/paper/2601.10323/) (2026)
- [SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenance](https://mlsanity.com/paper/2603.12703/) (2026)
- [StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMs](https://mlsanity.com/paper/2608.13416/) (2026)
- [Harnessing Streaming Video in the Wild](https://mlsanity.com/paper/2606.08615/) (2026)
- [FOLIO: Focused Semantic Memory for Streaming Video Understanding](https://mlsanity.com/paper/2607.13298/) (2026)
- [Realtime-Venus: A full-duplex interaction system with asynchronous delegation](https://mlsanity.com/paper/2609.13814/) (2026)
- [From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Models](https://mlsanity.com/paper/2603.04592/) (2026)
- [LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams](https://mlsanity.com/paper/2606.17798/) (2026)
- [InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos](https://mlsanity.com/paper/2608.01157/) (2026)
- [Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding](https://mlsanity.com/paper/2606.06991/) (2026)
- [Never Seen Before: Benchmarking Genuine Zero-Shot Composed Image Retrieval with Consistent Video-Sourced Datasets](https://mlsanity.com/paper/2606.07032/) (2026)
- [GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video](https://mlsanity.com/paper/2607.02991/) (2026)
- [OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction](https://mlsanity.com/paper/2610.01762/) (2026)
- [QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding](https://mlsanity.com/paper/2607.04559/) (2026)
- [Scaling Articulated Rationales for MLLM-based Recommendation](https://mlsanity.com/paper/2609.17639/) (2026)
- [Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams](https://mlsanity.com/paper/2609.27303/) (2026)
- [When Should the Count Change? Learning State Maintenance for Causal Video Counting](https://mlsanity.com/paper/2609.35416/) (2026)
- [Watch-Think-Interact: Bootstrapping Long-Horizon Multi-Turn Streaming Video Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2609.37035/) (2026)
- [Foresight: planning future perception in streaming VLMs without retraining](https://mlsanity.com/paper/2610.03123/) (2026)
- [InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems](https://mlsanity.com/paper/2610.05775/) (2026)
- [Proactive Physical AI through Continuous Observation and Timely Intervention](https://mlsanity.com/paper/2610.16967/) (2026)

## Abstract (arXiv)

Despite significant progress in Video Large Language Models (Video-LLMs) for offline video understanding, existing online Video-LLMs typically struggle to simultaneously process continuous frame-by-frame inputs and determine optimal response timing, often compromising real-time responsiveness and narrative coherence. To address these limitations, we introduce LiveStar, a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding. Specifically, LiveStar incorporates: (1) a training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) a response-silence decoding framework that determines optimal proactive response timing via a single forward pass verification; (3) memory-aware acceleration via peak-end memory compression for online inference on 10+ minute videos, combined with streaming key-value cache to achieve 1.53x faster inference. We also construct an OmniStar dataset, a comprehensive dataset for training and benchmarking that encompasses 15 diverse real-world scenarios and 5 evaluation tasks for online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar's state-of-the-art performance, achieving an average 19.5% improvement in semantic correctness with 18.1% reduced timing difference compared to existing online Video-LLMs, while improving FPS by 12.0% across all five OmniStar tasks. Our model and dataset can be accessed at https://github.com/yzy-bupt/LiveStar.