mlsanityData to 7 Oct 2026

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

arXiv 2311.17043 ↗First posted 28 Nov 2023cs.CV cs.CLcited by 539

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

Key facts. Cited by 539 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 73 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

48 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2023Visual Instruction Tuningcited by 7,792
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
35 more
  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2019Towards VQA Models That Can Readcited by 1,696
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  13. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  14. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  15. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  17. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  18. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  19. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  20. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  21. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  23. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  24. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  25. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  26. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  27. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  28. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  29. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  30. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  31. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  32. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  33. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  34. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  35. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 163; 2025: 252; 2026: 122.

539 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  3. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  5. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  6. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  7. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
  8. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  9. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  10. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  11. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  12. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
38 more
  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  3. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  4. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  5. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  6. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  7. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  8. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  9. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  10. 2023Video Understanding with Large Language Models: A Surveycited by 203
  11. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  12. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  13. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  14. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  15. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  16. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  17. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  18. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  19. 2024Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Taskscited by 146
  20. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  21. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  22. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  23. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  24. 2025OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video Understanding?cited by 134
  25. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  26. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  27. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  28. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  29. 2024VISA: Reasoning Video Object Segmentation via Large Language Modelscited by 123
  30. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  31. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  32. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 116
  33. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  34. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  35. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  36. 2025StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modelingcited by 107
  37. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  38. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103

50 most cited of 539 All 539 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
  2. 2024Long Context Transfer from Language to Vision
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
  4. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Models
  5. 2024LongVLM: Efficient Long Video Understanding via Large Language Models
  6. 2025D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition
  7. 2025MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
  8. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processing