mlsanityData to 7 Oct 2026

Video-LLaVA: Learning United Visual Representation by Alignment Before Projection

arXiv 2311.10122 ↗First posted 16 Nov 2023cs.CVcited by 1,385

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding. Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models. However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4 image benchmark toolkits. Additionally, our Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos. We aim for this work to provide modest insights into the multi-modal inputs for the LLM. Code address: hrefhttps://github.com/PKU-YuanGroup/Video-LLaVA

Key facts. Cited by 1,385 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 57 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

47 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
34 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  13. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  14. 2023PaLM 2 Technical Reportcited by 1,154
  15. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  16. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  17. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  18. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  21. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  22. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  23. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  24. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  25. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  26. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  27. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  28. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  29. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  30. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  31. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  32. 2023ImageBind-LLM: Multi-modality Instruction Tuningcited by 142
  33. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  34. 2020PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generationcited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 355; 2025: 626; 2026: 396.

1,385 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  5. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  6. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 398
  7. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  8. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  9. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  10. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  11. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  12. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
38 more
  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  3. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  4. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  5. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  6. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  7. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  9. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  10. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
  11. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  12. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  13. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  14. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  15. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  16. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  17. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  18. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  19. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  20. 2023Video Understanding with Large Language Models: A Surveycited by 203
  21. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  22. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  23. 2025VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interactioncited by 191
  24. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  25. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  26. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 178
  27. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  28. 2024An Introduction to Vision-Language Modelingcited by 175
  29. 2024TokenPacker: Efficient Visual Projector for Multimodal LLMcited by 170
  30. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  31. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  32. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  33. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  34. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 157
  35. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  36. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  37. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  38. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151

50 most cited of 1,385 All 1,385 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
  2. 2024TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerations
  3. 2025Delta-LLaVA: Base-then-Specialize Alignment for Token-Efficient Vision-Language Models
  4. 2025Rethinking Visual Information Processing in Multimodal LLMs
  5. 2023InfMLLM: A Unified Framework for Visual-Language Tasks
  6. 2024TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings
  7. 2025Visual Representation Alignment for Multimodal Large Language Models
  8. 2024LLaVA-KD: A Framework of Distilling Multimodal Large Language Models