# CogStream: Context-guided Streaming Video Question Answering

arXiv: https://arxiv.org/abs/2506.10516 · HTML: https://mlsanity.com/paper/2506.10516/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 30 of 38)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Long Context Transfer from Language to Vision](https://mlsanity.com/paper/2406.16852/) (2024)
- [COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysis](https://mlsanity.com/paper/1903.02874/) (2019)
- [Towards Real-Time Multi-Object Tracking](https://mlsanity.com/paper/1909.12605/) (2019)
- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams](https://mlsanity.com/paper/2406.08085/) (2024)
- [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024)
- [Streaming Video Question-Answering with In-context Video KV-Cache Retrieval](https://mlsanity.com/paper/2503.00540/) (2025)
- [Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge](https://mlsanity.com/paper/2501.13468/) (2025)
- [QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queries](https://mlsanity.com/paper/2107.09609/) (2021)
- [Koala: Key frame-conditioned long video-LLM](https://mlsanity.com/paper/2404.04346/) (2024)
- [SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding](https://mlsanity.com/paper/2502.10810/) (2025)

## Built on by (most cited first, 5 of 5)

- [OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoning](https://mlsanity.com/paper/2604.17052/) (2026)
- [Thinking in Streaming Video](https://mlsanity.com/paper/2603.12938/) (2026)
- [Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding](https://mlsanity.com/paper/2609.04131/) (2026)
- [From Priors to Perception: Grounding Video-LLMs in Physical Reality](https://mlsanity.com/paper/2605.04515/) (2026)
- [Companion-style QA Assistance in Ego-Vision](https://mlsanity.com/paper/2609.06721/) (2026)

## Abstract (arXiv)

Despite advancements in Video Large Language Models (Vid-LLMs) improving multimodal understanding, challenges persist in streaming video reasoning due to its reliance on contextual information. Existing paradigms feed all available historical contextual information into Vid-LLMs, resulting in a significant computational burden for visual data processing. Furthermore, the inclusion of irrelevant context distracts models from key details. This paper introduces a challenging task called Context-guided Streaming Video Reasoning (CogStream), which simulates real-world streaming video scenarios, requiring models to identify the most relevant historical contextual information to deduce answers for questions about the current stream. To support CogStream, we present a densely annotated dataset featuring extensive and hierarchical question-answer pairs, generated by a semi-automatic pipeline. Additionally, we present CogReasoner as a baseline model. It effectively tackles this task by leveraging visual stream compression and historical dialogue retrieval. Extensive experiments prove the effectiveness of this method.