mlsanityData to 14 Oct 2026

LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistant

arXiv 2503.03663 ↗First posted 5 Mar 2025cs.CVcited by 58

Not on the map. Why?

Abstract (arXiv)

First-person video assistants are highly anticipated to enhance our daily lives through online video dialogue. However, existing online video assistants often sacrifice assistant efficacy for real-time efficiency by processing low-frame-rate videos with coarse-grained visual features.To overcome the trade-off between efficacy and efficiency, we propose "Fast & Slow Video-Language Thinker" as an onLIne videO assistaNt, LION-FS, achieving real-time, proactive, temporally accurate, and contextually precise responses. LION-FS adopts a two-stage optimization strategy: 1)Fast Path: Routing-Based Response Determination evaluates frame-by-frame whether an immediate response is necessary. To enhance response determination accuracy and handle higher frame-rate inputs efficiently, we employ Token Aggregation Routing to dynamically fuse spatiotemporal features without increasing token numbers, while utilizing Token Dropping Routing to eliminate redundant features. 2)Slow Path: Multi-granularity Keyframe Augmentation optimizes keyframes during response generation. To provide comprehensive and detailed responses beyond atomic actions constrained by training data, fine-grained spatial features and human-environment interaction features are extracted through multi-granular pooling. These features are further integrated into a meticulously designed multimodal Thinking Template to guide more precise response generation. Comprehensive evaluations on online video tasks demonstrate that LION-FS achieves state-of-the-art efficacy and efficiency.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 62 of its 77 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

62 of 77 references.

References · 62

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2020End-to-End Object Detection with Transformerscited by 7,414
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
38 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  3. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  6. 2024Long Context Transfer from Language to Visioncited by 515
  7. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  8. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  9. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  10. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  11. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  12. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  13. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
  14. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  15. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  16. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  17. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 169
  18. 2024Streaming Long Video Understanding with Large Language Modelscited by 167
  19. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  20. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  21. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  22. 2020Multi-modal Dense Video Captioningcited by 113
  23. 2023Detecting and Grounding Multi-Modal Media Manipulationcited by 99
  24. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  25. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  26. 2024Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Taskscited by 84
  27. 2024Streaming Dense Video Captioningcited by 82
  28. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 81
  29. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluationcited by 81
  30. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  31. 2019Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baselinecited by 61
  32. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  33. 2023Detecting and Grounding Multi-Modal Media Manipulation and Beyondcited by 56
  34. 2020VD-BERT: A Unified Vision and Dialog Transformer with BERTcited by 55
  35. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 55
  36. 2024VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computationcited by 55
  37. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  38. 2024Visual Agents as Fast and Slow Thinkerscited by 53

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 36.

58 papers.

Sort:

By area:

  1. 2025JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigationcited by 89
  2. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  3. 2025CogVLA: Cognition-Aligned Vision-Language-Action Model via Instruction-Driven Routing & Sparsificationcited by 74
  4. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  5. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  6. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 46
  7. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agentcited by 35
  8. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  9. 2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 26
  10. 2025SwiftVLA: Unlocking Spatiotemporal Dynamics for Lightweight VLA Models at Minimal Overheadcited by 26
  11. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 25
  12. 2025SemanticVLA: Semantic-Aligned Sparsification and Enhancement for Efficient Robotic Manipulationcited by 21
38 more
  1. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skillscited by 19
  2. 2025UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queriescited by 18
  3. 2026Thinking in Streaming Videocited by 16
  4. 2025Cost-Effective Communication: An Auction-based Method for Language Agent Interactioncited by 14
  5. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  6. 2025PUMA: Layer-Pruned Language Model for Efficient Unified Multimodal Retrieval with Modality-Adaptive Learningcited by 11
  7. 2026DeltaVLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variationcited by 10
  8. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Modelscited by 8
  9. 2026Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understandingcited by 7
  10. 2026IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streamscited by 7
  11. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 6
  12. 2026ConsisVLA-4D: Advancing Spatiotemporal Consistency in Efficient 3D-Perception and 4D-Reasoning for Robotic Manipulationcited by 6
  13. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 6
  14. 2025Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanismcited by 4
  15. 2026STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understandingcited by 4
  16. 2026Harnessing Streaming Video in the Wildcited by 4
  17. 2026FOLIO: Focused Semantic Memory for Streaming Video Understandingcited by 4
  18. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  19. 2026TDEC: Deep Embedded Image Clustering with Transformer and Distribution Informationcited by 3
  20. 2026MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonalitycited by 3
  21. 2025Can Multi-Modal LLMs Provide Live Step-by-Step Task Guidance?cited by 2
  22. 2026Decouple and Cache: KV Cache Construction for Streaming Video Understandingcited by 2
  23. 2026Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understandingcited by 2
  24. 2026DynaTok: Temporally Adaptive and Positional Bias-Aware Token Compression for Video-LLMscited by 2
  25. 2026LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streamscited by 2
  26. 2025Automated Genomic Interpretation via Concept Bottleneck Models for Medical Roboticscited by 1
  27. 2025Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cachecited by 1
  28. 2026Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMUcited by 1
  29. 2026Interpretable Traffic Responsibility from Dashcam Video via Legal Multi Agent Reasoningcited by 1
  30. 2026CNMBI: Determining the Number of Clusters Using Center Pairwise Matching and Boundary Filteringcited by 1
  31. 2026Deep Image Clustering Based on Curriculum Learning and Density Informationcited by 1
  32. 2026FlowNar: Scalable Streaming Narration for Long-Form Videoscited by 1
  33. 2026Don't Pause: Streaming Video-Language Synchrony for Online Video Understandingcited by 1
  34. 2026Streaming Interventions: Can Video Large Language Models Correct Mistakes as They Occur?cited by 1
  35. 2026ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inferencecited by 1
  36. 2026StreamEMS: Streaming Video Understanding with Self-Evolving Memory Scheme for Vision-Language Modelscited by 1
  37. 2026UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modelingcited by 1
  38. 2026SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selectioncited by 1

50 most cited of 58 All 58 (JSON)