# Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2603.11896 · HTML: https://mlsanity.com/paper/2603.11896/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 12 Mar 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 33)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Video-R1: Reinforcing Video Reasoning in MLLMs](https://mlsanity.com/paper/2503.21776/) (2025)
- [LVBench: An Extreme Long Video Understanding Benchmark](https://mlsanity.com/paper/2406.08035/) (2024)
- [Self-attention Does Not Need O(n2) Memory](https://mlsanity.com/paper/2112.05682/) (2021)
- [Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognition](https://mlsanity.com/paper/2501.03230/) (2024)
- [StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding](https://mlsanity.com/paper/2411.03628/) (2024)
- [TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos](https://mlsanity.com/paper/2504.17343/) (2025)
- [Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory](https://mlsanity.com/paper/2508.09736/) (2025)
- [StreamingVLM: Real-Time Understanding for Infinite Video Streams](https://mlsanity.com/paper/2510.09608/) (2025)
- [Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning](https://mlsanity.com/paper/2508.04416/) (2025)
- [Scaling RL to Long Videos](https://mlsanity.com/paper/2507.07966/) (2025)
- [Flash-VStream: Efficient Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2506.23825/) (2025)
- [StreamForest: Efficient Online Video Understanding with Persistent Event Memory](https://mlsanity.com/paper/2509.24871/) (2025)
- [LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrieval](https://mlsanity.com/paper/2505.15269/) (2025)
- [StreamingTOM: Streaming Token Compression for Efficient Video Understanding](https://mlsanity.com/paper/2510.18269/) (2025)
- [ViSpeak: Visual Instruction Feedback in Streaming Videos](https://mlsanity.com/paper/2503.12769/) (2025)
- [Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm](https://mlsanity.com/paper/2511.04570/) (2025)
- [StreamAgent: Towards Anticipatory Agents for Streaming Video Understanding](https://mlsanity.com/paper/2508.01875/) (2025)
- [Eyes Wide Open: Ego Proactive Video-LLM for Streaming Video](https://mlsanity.com/paper/2510.14560/) (2025)
- [StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression](https://mlsanity.com/paper/2511.07278/) (2025)
- [LiveStar: Live Streaming Assistant for Real-World Online Video Understanding](https://mlsanity.com/paper/2511.05299/) (2025)
- [StreamChat: Chatting with Streaming Video](https://mlsanity.com/paper/2412.08646/) (2024)
- [RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video](https://mlsanity.com/paper/2505.02064/) (2025)
- [StreamingThinker: Large Language Models Can Think While Reading](https://mlsanity.com/paper/2510.17238/) (2025)
- [StreamEQA: Towards Streaming Video Understanding for Embodied Scenarios](https://mlsanity.com/paper/2512.04451/) (2025)
- [StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos](https://mlsanity.com/paper/2512.01707/) (2025)
- [Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2601.06843/) (2026)

## Built on by (most cited first, 8 of 8)

- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do](https://mlsanity.com/paper/2606.22565/) (2026)
- [FOLIO: Focused Semantic Memory for Streaming Video Understanding](https://mlsanity.com/paper/2607.13298/) (2026)
- [Omni-Streaming Thinking](https://mlsanity.com/paper/2609.15128/) (2026)
- [OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interaction](https://mlsanity.com/paper/2610.01762/) (2026)
- [TimeInteract: Towards Real-Time Interactive Intelligence for Streaming Time Series](https://mlsanity.com/paper/2609.26389/) (2026)
- [APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants](https://mlsanity.com/paper/2609.37559/) (2026)
- [DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding](https://mlsanity.com/paper/2610.19849/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: https://github.com/wl666hhh/Think_While_Watching/