mlsanityData to 7 Oct 2026

SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models

arXiv 2407.15841 ↗First posted 22 Jul 2024cs.CVcited by 123

Not on the map. Why?

Cited by 69 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video large language model (LLM) that can jointly capture detailed spatial semantics and long-range temporal context without exceeding the token budget of commonly used LLMs. This is realized by using a two-stream SlowFast design of inputs for Video LLMs to aggregate features from sampled frames in an effective way. Specifically, the Slow pathway extracts features at a low frame rate while keeping as much spatial detail as possible (e.g., with 12x24 tokens), and the Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As a result, this design allows us to adequately capture both spatial and temporal features that are beneficial for detailed video understanding. Experimental results show that SF-LLaVA outperforms existing training-free methods on a wide range of video tasks. On some benchmarks, it achieves comparable or even better performance compared to state-of-the-art Video LLMs that are fine-tuned on video datasets. Code has been made available at: https://github.com/apple/ml-slowfast-llava.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 39 of its 46 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

39 of 46 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2018SlowFast Networks for Video Recognitioncited by 1,862
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  12. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
27 more
  1. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  2. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  6. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  8. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  9. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  10. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  11. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  12. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  13. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  14. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  15. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  16. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  17. 2016TGIF: A New Dataset and Benchmark on Animated GIF Descriptioncited by 124
  18. 20234M: Massively Multimodal Masked Modelingcited by 118
  19. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  20. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  21. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  22. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  23. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
  24. 2023Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokenscited by 70
  25. 2024Language Repository for Long Video Understandingcited by 55
  26. 20244M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalitiescited by 52
  27. 2024FreeVA: Offline MLLM as Training-Free Video Assistantcited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 26; 2025: 68; 2026: 29.

123 papers.

Sort:

By area:

  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  5. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
  6. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  7. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  8. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  9. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  10. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  11. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 87
  12. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
38 more
  1. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  2. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  3. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 72
  4. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  5. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Modelscited by 64
  6. 2025LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understandingcited by 57
  7. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  8. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  9. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  10. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  11. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  12. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 38
  13. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  14. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  15. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  16. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  17. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  18. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  19. 2025AToken: A Unified Tokenizer for Visioncited by 30
  20. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  21. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  22. 2025SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understandingcited by 25
  23. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QAcited by 24
  24. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  25. 2025LongVideoAgent: Multi-Agent Reasoning with Long Videoscited by 23
  26. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  27. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understandingcited by 21
  28. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  29. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  30. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  31. 2025Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Modelscited by 19
  32. 2025Breaking Down Video LLM Benchmarks: Knowledge, Spatial Perception, or True Temporal Understanding?cited by 19
  33. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  34. 2025UniVid: The Open-Source Unified Video Modelcited by 18
  35. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  36. 2025Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Markscited by 17
  37. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  38. 2025MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understandingcited by 16

50 most cited of 123 All 123 (JSON)