mlsanityData to 7 Oct 2026

LLaVA-Video: Video Instruction Tuning With Synthetic Data

arXiv 2410.02713 ↗First posted 3 Oct 2024cs.CV cs.CLcited by 550

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.

Key facts. Cited by 550 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 66 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

52 of 66 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  2. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  3. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward1 Apr
  4. 2023VILA: On Pre-training for Visual Language Models12 Dec

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018SlowFast Networks for Video Recognitioncited by 1,862
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
36 more
  1. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  2. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  8. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  9. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  10. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  11. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  12. 2024Long Context Transfer from Language to Visioncited by 502
  13. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  14. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  15. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  16. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  17. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  18. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  19. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  20. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  21. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  22. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  23. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  24. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  25. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  26. 2015A Dataset for Movie Descriptioncited by 195
  27. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  28. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  29. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 131
  30. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  31. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  32. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  33. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  34. 2023Shot2Story: A New Benchmark for Comprehensive Understanding of Multi-shot Videoscited by 54
  35. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  36. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 238; 2026: 275.

550 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  5. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  6. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 248
  7. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  8. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  9. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  10. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  11. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  12. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 147
38 more
  1. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  2. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 131
  3. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 125
  4. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  5. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 111
  6. 2025Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?cited by 107
  7. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 101
  8. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  9. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  10. 2025Visual Spatial Tuningcited by 85
  11. 2025Scaling RL to Long Videoscited by 83
  12. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  13. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 79
  14. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  15. 2024Multimodal Alignment and Fusion: A Surveycited by 73
  16. 2025LongCat-Video Technical Reportcited by 73
  17. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  18. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  19. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 69
  20. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  21. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 66
  22. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  23. 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
  24. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  25. 2025Ming-Omni: A Unified Multimodal Model for Perception and Generationcited by 62
  26. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoningcited by 59
  27. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  28. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  29. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  30. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  31. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  32. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  33. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  34. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 46
  35. 2025AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understandingcited by 45
  36. 2024VideoLLM Knows When to Speak: Enhancing Time-Sensitive Video Comprehension with Video-Text Duet Interaction Formatcited by 44
  37. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  38. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41

50 most cited of 550 All 550 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improved Baselines with Visual Instruction Tuning
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  4. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
  5. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  6. 2026Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios
  7. 2025InstructionBench: An Instructional Video Understanding Benchmark
  8. 2024Sparrow: Data-Efficient Video-LLM with Text-to-Image Augmentation