# Slot-VLM: SlowFast Slots for Video-Language Modeling

arXiv: https://arxiv.org/abs/2402.13088 · HTML: https://mlsanity.com/paper/2402.13088/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 20 Feb 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [MONet: Unsupervised Scene Decomposition and Representation](https://mlsanity.com/paper/1901.11390/) (2019)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [Bridging the Gap to Real-World Object-Centric Learning](https://mlsanity.com/paper/2209.14860/) (2022)
- [Illiterate DALL-E Learns to Compose](https://mlsanity.com/paper/2110.11405/) (2021)
- [ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst](https://mlsanity.com/paper/2305.16103/) (2023)
- [BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning](https://mlsanity.com/paper/2309.15785/) (2023)

## Built on by (most cited first, 8 of 8)

- [Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters](https://mlsanity.com/paper/2411.03312/) (2024)
- [MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representations](https://mlsanity.com/paper/2503.15871/) (2025)
- [VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges](https://mlsanity.com/paper/2409.01071/) (2024)
- [Zero-Shot Object-Centric Representation Learning](https://mlsanity.com/paper/2408.09162/) (2024)
- [VideoOrion: Tokenizing Object Dynamics in Videos](https://mlsanity.com/paper/2411.16156/) (2024)
- [Object-Centric Pretraining via Target Encoder Bootstrapping](https://mlsanity.com/paper/2503.15141/) (2025)
- [DisCo: Towards Distinct and Coherent Visual Encapsulation in Video MLLMs](https://mlsanity.com/paper/2507.10302/) (2025)
- [Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey](https://mlsanity.com/paper/2604.11283/) (2026)

## Abstract (arXiv)

Video-Language Models (VLMs), powered by the advancements in Large Language Models (LLMs), are charting new frontiers in video understanding. A pivotal challenge is the development of an efficient method to encapsulate video content into a set of representative tokens to align with LLMs. In this work, we introduce Slot-VLM, a novel framework designed to generate semantically decomposed video tokens, in terms of object-wise and event-wise visual representations, to facilitate LLM inference. Particularly, we design a SlowFast Slots module, i.e., SF-Slots, that adaptively aggregates the dense video tokens from the CLIP vision encoder to a set of representative slots. In order to take into account both the spatial object details and the varied temporal dynamics, SF-Slots is built with a dual-branch structure. The Slow-Slots branch focuses on extracting object-centric slots from features at high spatial resolution but low (slow) frame sample rate, emphasizing detailed object information. Conversely, Fast-Slots branch is engineered to learn event-centric slots from high temporal sample rate but low spatial resolution features. These complementary slots are combined to form the vision context, serving as the input to the LLM for efficient question answering. Our experimental results demonstrate the effectiveness of our Slot-VLM, which achieves the state-of-the-art performance on video question-answering.