mlsanityData to 14 Oct 2026

xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs

arXiv 2410.16267 ↗First posted 21 Oct 2024cs.CV cs.AI cs.CLcited by 34

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens. This enables BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs. 4608 tokens). We explore different types of temporal encoders, including learnable spatio-temporal pooling as well as sequential models like Token Turing Machines. We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using fewer visual tokens. The project website is at https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 51 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

46 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  3. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  4. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  11. 2022Token Merging: Your ViT But Fastercited by 1,021
  12. 2023VideoChat: Chat-Centric Video Understandingcited by 857
27 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 642
  3. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  4. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  5. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  6. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  7. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  8. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  9. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  10. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  11. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  13. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  14. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  15. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  16. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  17. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 121
  18. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  19. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  20. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 95
  21. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  22. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  23. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 50
  24. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  25. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  26. 2022Token Turing Machinescited by 25
  27. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 16; 2026: 11.

34 papers.

Sort:

By area:

  1. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 96
  2. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 49
  3. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  4. 2025VideoHallu: Evaluating and Mitigating Multi-modal Hallucinations on Synthetic Video Understandingcited by 32
  5. 2024InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Modelscited by 31
  6. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 31
  7. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  8. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 29
  9. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 28
  10. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  11. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  12. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
22 more
  1. 2024LinVT: Empower Your Image-level Large Language Model to Understand Videoscited by 19
  2. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  3. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 13
  4. 2025LLaVA-Octopus: Unlocking Instruction-Driven Adaptive Projector Fusion for Video Understandingcited by 10
  5. 2024freePruner: A Training-free Approach for Large Multimodal Model Accelerationcited by 8
  6. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  7. 2025DisTime: Distribution-based Time Representation for Video Large Language Modelscited by 7
  8. 2025DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understandingcited by 7
  9. 2025Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Datacited by 6
  10. 2026VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latentscited by 3
  11. 2026Video-Based Reward Modeling for Computer-Use Agentscited by 3
  12. 2025VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Modelscited by 2
  13. 2025NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Modelscited by 2
  14. 2026Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unificationcited by 2
  15. 2026HiVid-Narrator: Hierarchical Video Narrative Generation with Scene-Primed ASR-anchored Compressioncited by 1
  16. 2026Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMscited by 1
  17. 2026Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddingscited by 1
  18. 2026TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Modelscited by 0
  19. 2026Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QAcited by 0
  20. 2026Efficient Tracking and Understanding Object Transformationscited by 0
  21. 2026StateTrace: An Object-Centric Framework for Hidden-State Spatiotemporal Reasoning in Long Videoscited by 0
  22. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Video Panels for Long Video Understanding
  2. 2025An Empirical Study for Representations of Videos in Video Question Answering via MLLMs
  3. 2026Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding
  4. 2025MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
  5. 2025Test-Time Temporal Sampling for Efficient MLLM Video Understanding
  6. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models
  7. 2024Lost in Time: A New Temporal Benchmark for VideoLLMs
  8. 2025Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models