mlsanityData to 14 Oct 2026

TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos

arXiv 2504.17343 ↗First posted 24 Apr 2025cs.CVcited by 97

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid growth of online video platforms, particularly live streaming services, has created an urgent need for real-time video understanding systems. These systems must process continuous video streams and respond to user queries instantaneously, presenting unique challenges for current Video Large Language Models (VideoLLMs). While existing VideoLLMs excel at processing complete videos, they face significant limitations in streaming scenarios due to their inability to handle dense, redundant frames efficiently. We introduce TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video interaction. At its core lies our innovative Differential Token Drop (DTD) module, which addresses the fundamental challenge of visual redundancy in streaming videos. Drawing inspiration from human visual perception's Change Blindness phenomenon, DTD preserves meaningful temporal changes while filtering out static, redundant content between frames. Remarkably, our experiments demonstrate that DTD achieves an 82.8% reduction in video tokens while maintaining 98% performance on StreamingBench, revealing that over 80% of visual content in streaming videos is naturally redundant without requiring language guidance. To enable seamless real-time interaction, we present TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse interaction patterns including backward-tracing, current-perception, and future-responding scenarios. TimeChat-Online's unique Proactive Response capability, naturally achieved through continuous monitoring of video scene transitions via DTD, sets it apart from conventional approaches. Our extensive evaluation demonstrates TimeChat-Online's superior performance on streaming benchmarks (StreamingBench and OvOBench) and maintaining competitive results on long-form video tasks such as Video-MME and MLVU.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Builds on 49 of its 75 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

49 of 75 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  12. 2025Qwen2.5-Omni Technical Reportcited by 736
37 more
  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  2. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  4. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  6. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  8. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  9. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  10. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  11. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  12. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  13. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  14. 2025Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters Morecited by 155
  15. 2024StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understandingcited by 144
  16. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 142
  17. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  18. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  19. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 118
  20. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 117
  21. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98
  22. 2024Is Cosine-Similarity of Embeddings Really About Similarity?cited by 89
  23. 2023Hierarchical Video-Moment Retrieval and Step-Captioningcited by 83
  24. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  25. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  26. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  27. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 70
  28. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 59
  29. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  30. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  31. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  32. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 47
  33. 2024Don't Look Twice: Faster Video Transformers with Run-Length Tokenizationcited by 43
  34. 2024Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videoscited by 38
  35. 2024Temporal Reasoning Transfer from Text to Videocited by 38
  36. 2023Movie101: A New Movie Understanding Benchmarkcited by 25
  37. 2024Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPUcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 80.

97 papers.

Sort:

By area:

  1. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  2. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  3. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  4. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 43
  5. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 41
  6. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 38
  7. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 37
  8. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  9. 2026A Simple Baseline for Streaming Video Understandingcited by 33
  10. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 28
  11. 2025ProactiveVideoQA: A Comprehensive Benchmark Evaluating Proactive Interactions in Video Large Language Modelscited by 27
  12. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
38 more
  1. 2026OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Modelscited by 25
  2. 2025MMDuet2: Enhancing Proactive Interaction of Video MLLMs with Multi-Turn Reinforcement Learningcited by 22
  3. 2026Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneouslycited by 22
  4. 2026StreamReady: Learning What to Answer and When in Long Streaming Videoscited by 19
  5. 2026AURA: Always-On Understanding and Real-Time Assistance via Video Streamscited by 16
  6. 2026Proact-VL: A Proactive VideoLLM for Real-Time AI Companionscited by 15
  7. 2026JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligencecited by 14
  8. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 12
  9. 2026EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Usecited by 12
  10. 2025VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMscited by 11
  11. 2026TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captionscited by 11
  12. 2025StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understandingcited by 10
  13. 2026VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understandingcited by 10
  14. 2026Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Modelcited by 10
  15. 2026PEARL: Personalized Streaming Video Understanding Modelcited by 9
  16. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  17. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Modelscited by 8
  18. 2026Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Modelscited by 8
  19. 2026Semantic-Aware Adaptive Visual Memory for Streaming Video Understandingcited by 8
  20. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  21. 2026What Should a Streaming Video Model Remember?cited by 7
  22. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 6
  23. 2026OVO-S-Bench: A Hierarchical Benchmark for Streaming Spatial Intelligence in Multimodal LLMscited by 6
  24. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 6
  25. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  26. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencecited by 5
  27. 2026Progressive Online Video Understanding with Evidence-Aligned Timing and Transparent Decisionscited by 5
  28. 2026EvoStreaming: Your Offline Video Model Is a Natively Streaming Assistantcited by 5
  29. 2026StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggeringcited by 5
  30. 2025EgoPrune: Efficient Token Pruning for Egomotion Video Reasoning in Embodied Agentcited by 4
  31. 2026StreamingClaw Technical Reportcited by 4
  32. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 4
  33. 2026Don't Pause! Every prediction matters in a streaming videocited by 4
  34. 2026StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Videocited by 4
  35. 2026An Efficient Streaming Video Understanding Framework with Agentic Controlcited by 4
  36. 2026Harnessing Streaming Video in the Wildcited by 4
  37. 2026Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understandingcited by 4
  38. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4

50 most cited of 97 All 97 (JSON)