mlsanityData to 14 Oct 2026

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

arXiv 2306.02858 ↗First posted 5 Jun 2023cs.CL cs.CV cs.SDcited by 1,471

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

We present Video-LLaMA a multi-modal framework that empowers Large Language Models (LLMs) with the capability of understanding both visual and auditory content in the video. Video-LLaMA bootstraps cross-modal training from the frozen pre-trained visual and audio encoders and the frozen LLMs. Unlike previous works that complement LLMs to process the visual or audio signals only, Video-LLaMA enables video comprehension by tackling two challenges: (1) capturing the temporal changes in visual scenes, (2) integrating audio-visual signals. To counter the first challenge, we propose a Video Q-former to assemble a pre-trained image encoder into our video encoder and introduce a video-to-text generation task to learn video-language correspondence. For the second challenge, we leverage ImageBind, a universal embedding model aligning multiple modalities, as the pre-trained audio encoder and introduce an Audio Q-former on top of ImageBind to learn reasonable auditory query embeddings for the LLM module. To align the output of both visual and audio encoders with LLM's embedding space, we first train Video-LLaMA on massive video/image-caption pairs and then tune our model with visual-instruction datasets of moderate amount but higher quality. We found Video-LLaMA shows the ability to perceive and comprehend video content and generate meaningful responses grounded in the visual and auditory information presented in the videos.

Key facts. Cited by 1,471 papers in the map (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 41 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

32 of 41 references.

References · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
20 more
  1. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  2. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  8. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  9. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  10. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  11. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  12. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  13. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  14. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  15. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 334
  16. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  17. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  18. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  19. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  20. 2023VPGTrans: Transfer Visual Prompt Generator across LLMscited by 80

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 112; 2024: 505; 2025: 551; 2026: 303.

1,471 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  7. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  8. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  9. 2023A Survey on Multimodal Large Language Modelscited by 909
  10. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
38 more
  1. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  2. 2023A Comprehensive Overview of Large Language Modelscited by 648
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  4. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  5. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  6. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  7. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  10. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  11. 2024Long Context Transfer from Language to Visioncited by 515
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
  13. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  14. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  15. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  16. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 402
  17. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  18. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  19. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  20. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  21. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  22. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  23. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  24. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  25. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 294
  26. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 292
  27. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  28. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  29. 2024Knowledge Conflicts for LLMs: A Surveycited by 283
  30. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  31. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  32. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  33. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  34. 2023FireAct: Toward Language Agent Fine-tuningcited by 256
  35. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  36. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  37. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  38. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234

50 most cited of 1,471 All 1,471 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learners
  2. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  3. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  4. 2025OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs
  5. 2025Unleashing Hour-Scale Video Training for Long Video-Language Understanding
  6. 2024Visual Context Window Extension: A New Perspective for Long Video Understanding
  7. 2025UFVideo: Towards Unified Fine-Grained Video Cooperative Understanding with Large Language Models
  8. 2026Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios