mlsanityData to 14 Oct 2026

StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition

arXiv 2503.06220 ↗First posted 8 Mar 2025cs.CV cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

With the rise of real-world human-AI interaction applications, such as AI assistants, the need for Streaming Video Dialogue is critical. To address this need, we introduce StreamMind, a video LLM framework that achieves ultra-FPS streaming video processing (100 fps on a single A100) and enables proactive, always-on responses in real time, without explicit user intervention. To solve the key challenge of the contradiction between linear video streaming speed and quadratic transformer computation cost, we propose a novel perception-cognition interleaving paradigm named ''event-gated LLM invocation'', in contrast to the existing per-time-step LLM invocation. By introducing a Cognition Gate network between the video encoder and the LLM, LLM is only invoked when relevant events occur. To realize the event feature extraction with constant cost, we propose Event-Preserving Feature Extractor (EPFE) based on state-space method, generating a single perception token for spatiotemporal features. These techniques enable the video LLM with full-FPS perception and real-time cognition response. Experiments on Ego4D and SoccerNet streaming tasks, as well as standard offline benchmarks, demonstrate state-of-the-art performance in both model capability and real-time efficiency, paving the way for ultra-high-FPS applications, such as Game AI and interactive media. The code and data is available at https://aka.ms/StreamMind.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 67 of its 78 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

67 of 78 references.

References · 67

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  9. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  12. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
38 more
  1. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  4. 2020Designing Network Design Spacescited by 960
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  6. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  8. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  9. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  11. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  13. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  14. 2024Long Context Transfer from Language to Visioncited by 515
  15. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 439
  16. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  17. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  18. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  19. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  20. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  21. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  22. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  23. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  24. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  25. 2019Connecting Vision and Language with Localized Narrativescited by 226
  26. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  27. 2022Learning Video Representations from Large Language Modelscited by 205
  28. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 197
  29. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  30. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  31. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  32. 2018SoccerNet: A Scalable Dataset for Action Spotting in Soccer Videoscited by 150
  33. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  34. 2022MatchFormer: Interleaving Attention in Transformers for Feature Matchingcited by 126
  35. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 125
  36. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  37. 2022MeMOT: Multi-Object Tracking with Memorycited by 114
  38. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 37.

46 papers.

Sort:

By area:

  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  2. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  3. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  4. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  5. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 26
  6. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  7. 2026Thinking in Streaming Videocited by 16
  8. 2026Proact-VL: A Proactive VideoLLM for Real-Time AI Companionscited by 15
  9. 2025Multi-RAG: A Multimodal Retrieval-Augmented Generation System for Adaptive Video Understandingcited by 13
  10. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
  11. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  12. 2026IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streamscited by 7
34 more
  1. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggeringcited by 5
  2. 2026StreamTTT: Reconciling Real-Time Perception and Long-Term Memory in Streaming VLMscited by 5
  3. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 4
  4. 2026Harnessing Streaming Video in the Wildcited by 4
  5. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  6. 2026WeaveTime: Stream from Earlier Frames into Emergent Memory in VideoLLMscited by 3
  7. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  8. 2026Visual Agentic Memory: Enabling Online Long Video Understanding via Online Indexing, Hierarchical Memory, and Agentic Retrievalcited by 3
  9. 2025Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?cited by 2
  10. 2025Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devicescited by 2
  11. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  12. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMscited by 2
  13. 2026LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streamscited by 2
  14. 2026ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandingcited by 2
  15. 2026On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understandingcited by 1
  16. 2026Don't Pause: Streaming Video-Language Synchrony for Online Video Understandingcited by 1
  17. 2026Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?cited by 1
  18. 2026ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inferencecited by 1
  19. 2026GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Videocited by 1
  20. 2026EgoPlay: Event-Triggered Video Editing for Egocentric Streamscited by 1
  21. 2026StreamEMS: Streaming Video Understanding with Self-Evolving Memory Scheme for Vision-Language Modelscited by 1
  22. 2026Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understandingcited by 1
  23. 2026OneStreamer: Unifying Perception, Memory, and Proactive Response in Streaming Video Interactioncited by 1
  24. 2025TrajSV: A Trajectory-based Model for Sports Video Representations and Applicationscited by 0
  25. 2026Zoom In, Reason Out: Efficient Far-field Anomaly Detection in Expressway Surveillance Videos via Focused VLM Reasoning Guided by Bayesian Inferencecited by 0
  26. 2026MASRA: MLLM-Assisted Semantic-Relational Consistent Alignment for Video Temporal Groundingcited by 0
  27. 2026NBA_Streaming: A Large-Scale Benchmark for Fine-Grained Basketball Commentary Generation in Continuous Streamscited by 0
  28. 2026A Survey of Large Models in Sportscited by 0
  29. 2026ReactVAU: A Slow-Fast Decoupled Framework for Streaming Video Anomaly Understandingcited by 0
  30. 2026Language-Augmented Video Action Anticipation: Design Fundamentals, Benchmarks, and Open Challengescited by 0
  31. 2026KeyRec: Bounded Visual Memory for Streaming and Long-Video Understandingcited by 0
  32. 2026LLMs are General Asynchronous Agentscited by 0
  33. 2026Foresight: planning future perception in streaming VLMs without retrainingcited by 0
  34. 2026Proactive Physical AI through Continuous Observation and Timely Interventioncited by 0