# Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

arXiv: https://arxiv.org/abs/2609.27303 · HTML: https://mlsanity.com/paper/2609.27303/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 51 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 30 of 40)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?](https://mlsanity.com/paper/2501.05510/) (2025)
- [Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction](https://mlsanity.com/paper/2501.03218/) (2025)
- [Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaboration](https://mlsanity.com/paper/2305.13626/) (2023)
- [StreamingVLM: Real-Time Understanding for Infinite Video Streams](https://mlsanity.com/paper/2510.09608/) (2025)
- [A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects](https://mlsanity.com/paper/2305.02750/) (2023)
- [StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant](https://mlsanity.com/paper/2505.05467/) (2025)
- [Online Video Understanding: OVBench and VideoChat-Online](https://mlsanity.com/paper/2501.00584/) (2024)
- [SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding](https://mlsanity.com/paper/2502.10810/) (2025)
- [LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval](https://mlsanity.com/paper/2505.15269/) (2025)
- [OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contexts](https://mlsanity.com/paper/2503.22952/) (2025)
- [LiveStar: Live Streaming Assistant for Real-World Online Video Understanding](https://mlsanity.com/paper/2511.05299/) (2025)
- [MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2512.06810/) (2025)
- [StreamReady: Learning What to Answer and When in Long Streaming Videos](https://mlsanity.com/paper/2603.08620/) (2026)
- [Thinking in Streaming Video](https://mlsanity.com/paper/2603.12938/) (2026)
- [An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue](https://mlsanity.com/paper/2501.16643/) (2025)
- [LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts](https://mlsanity.com/paper/1809.04938/) (2018)
- [Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach](https://mlsanity.com/paper/2501.17258/) (2025)
- [LiViBench: An Omnimodal Benchmark for Interactive Livestream Video Understanding](https://mlsanity.com/paper/2601.15016/) (2026)
- [KuaiLive: A Real-time Interactive Dataset for Live Streaming Recommendation](https://mlsanity.com/paper/2508.05633/) (2025)
- [ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding](https://mlsanity.com/paper/2601.10323/) (2026)

## Abstract (arXiv)

Livestreams are long-lasting interactive environments where audiovisual content, viewer activity, host behavior, and platform signals evolve together, creating assistance needs that emerge from the stream itself. We introduce liveassistant, a framework for mixed-initiative, role-conditioned assistance that formulates livestream interaction as four coupled decisions: whether to act, when to act, whom to address, and what to communicate. At each 10-second interval, one autoregressive policy consumes native audio and video with synchronized comments, gifts, viewer dynamics, and room metadata, then selects textscOBS, textscMEM, or textscANS. textscOBS remains silent, textscMEM records a private semantic update, and textscANS specifies a recipient, task, and grounded message. To support this task, we build a trajectory engine that reconstructs real livestream sessions into structured causal supervision, yielding over 320 hours of optimization trajectories and a human-reviewed benchmark of 275 clips and 13,812 decision intervals. We train the policy with Marker-Aware Multiturn Supervised Fine-Tuning (MA-MSFT), which strengthens sparse structured decisions, followed by Streaming Multiturn GSPO (SM-GSPO), which optimizes self-generated trajectories with turn- and trajectory-level credit. On the held-out benchmark, liveassistant reaches 71.14 state accuracy, 72.67 recipient accuracy, and 58.41 task accuracy, with consistent gains over representative streaming and general multimodal baselines. Together, the formulation, benchmark, and training framework establish livestream assistance as selective participation in a shared social stream.