mlsanityData to 7 Oct 2026

Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction

arXiv 2501.03218 ↗First posted 6 Jan 2025cs.CVcited by 122

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Active Real-time interaction with video LLMs introduces a new paradigm for human-computer interaction, where the model not only understands user intent but also responds while continuously processing streaming video on the fly. Unlike offline video LLMs, which analyze the entire video before answering questions, active real-time interaction requires three capabilities: 1) Perception: real-time video monitoring and interaction capturing. 2) Decision: raising proactive interaction in proper situations, 3) Reaction: continuous interaction with users. However, inherent conflicts exist among the desired capabilities. The Decision and Reaction require a contrary Perception scale and grain, and the autoregressive decoding blocks the real-time Perception and Decision during the Reaction. To unify the conflicted capabilities within a harmonious system, we present Dispider, a system that disentangles Perception, Decision, and Reaction. Dispider features a lightweight proactive streaming video processing module that tracks the video stream and identifies optimal moments for interaction. Once the interaction is triggered, an asynchronous interaction module provides detailed responses, while the processing module continues to monitor the video in the meantime. Our disentangled and asynchronous design ensures timely, contextually accurate, and computationally efficient responses, making Dispider ideal for active real-time interaction for long-duration video streams. Experiments show that Dispider not only maintains strong performance in conventional video QA tasks, but also significantly surpasses previous online models in streaming scenario responses, thereby validating the effectiveness of our architecture. The code and model are released at urlhttps://github.com/Mark12Ding/Dispider.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 56 of its 69 references that are in the map. First posted on arXiv 6 Jan 2025.

Builds on

56 of 69 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2023Mistral 7Bcited by 3,065
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  11. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  12. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  13. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  14. 2024Long Context Transfer from Language to Visioncited by 502
  15. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  16. 2024InternLM2 Technical Reportcited by 391
  17. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  18. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  19. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  20. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  21. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  22. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  23. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  24. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  25. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  26. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  27. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  28. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 172
  29. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  30. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  31. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  32. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  33. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  34. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  35. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  36. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  37. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  38. 2023Grounded Question-Answering in Long Egocentric Videoscited by 62

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 92.

122 papers.

Sort:

By area:

  1. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  2. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  3. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
  4. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  5. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  6. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  7. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  8. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  9. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  10. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  11. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  12. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
38 more
  1. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  2. 2026A Simple Baseline for Streaming Video Understandingcited by 29
  3. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
  4. 2025Streaming Video Instruction Tuningcited by 25
  5. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 24
  6. 2025Advancing Complex Video Object Segmentation via Progressive Concept Constructioncited by 24
  7. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
  8. 2025RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Videocited by 21
  9. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 19
  10. 2025AI for Service: Proactive Assistance with AI Glassescited by 18
  11. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 18
  12. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 18
  13. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 15
  14. 2026Thinking in Streaming Videocited by 14
  15. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 14
  16. 2026Proact-VL: A Proactive VideoLLM for Real-Time AI Companionscited by 12
  17. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 11
  18. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  19. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 10
  20. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 10
  21. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 8
  22. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  23. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 8
  24. 2025LiveCC: Learning Video LLM with Streaming Speech Transcription at Scalecited by 7
  25. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 7
  26. 2026What Should a Streaming Video Model Remember?cited by 7
  27. 2025AViLA: Asynchronous Vision-Language Agent for Streaming Multimodal Data Interactioncited by 6
  28. 2026Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Modelscited by 6
  29. 2026SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenancecited by 6
  30. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
  31. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  32. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 6
  33. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  34. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 5
  35. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  36. 2026Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisionscited by 5
  37. 2026EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistantcited by 5
  38. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4

50 most cited of 122 All 122 (JSON)