mlsanityData to 7 Oct 2026

VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding

arXiv 2501.13106 ↗First posted 22 Jan 2025cs.CVcited by 502

Not on the map. Why?

Cited by 206 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we propose VideoLLaMA3, a more advanced multimodal foundation model for image and video understanding. The core design philosophy of VideoLLaMA3 is vision-centric. The meaning of "vision-centric" is two-fold: the vision-centric training paradigm and vision-centric framework design. The key insight of our vision-centric training paradigm is that high-quality image-text data is crucial for both image and video understanding. Instead of preparing massive video-text datasets, we focus on constructing large-scale and high-quality image-text datasets. VideoLLaMA3 has four training stages: 1) Vision Encoder Adaptation, which enables vision encoder to accept images of variable resolutions as input; 2) Vision-Language Alignment, which jointly tunes the vision encoder, projector, and LLM with large-scale image-text data covering multiple types (including scene images, documents, charts) as well as text-only data. 3) Multi-task Fine-tuning, which incorporates image-text SFT data for downstream tasks and video-text data to establish a foundation for video understanding. 4) Video-centric Fine-tuning, which further improves the model's capability in video understanding. As for the framework design, to better capture fine-grained details in images, the pretrained vision encoder is adapted to encode images of varying sizes into vision tokens with corresponding numbers, rather than a fixed number of tokens. For video inputs, we reduce the number of vision tokens according to their similarity so that the representation of videos will be more precise and compact. Benefit from vision-centric designs, VideoLLaMA3 achieves compelling performances in both image and video understanding benchmarks.

Key facts. Cited by 502 papers in the map (data through 7 Oct 2026). Builds on 132 of its 157 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

132 of 157 references.

References · 132

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  13. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  15. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  17. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  18. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  19. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  20. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  21. 2016A Diagram Is Worth A Dozen Imagescited by 793
  22. 2017Dense-Captioning Events in Videoscited by 789
  23. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  24. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  25. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  26. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  27. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  28. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  29. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  30. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  31. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  32. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  33. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  34. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  35. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  36. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  37. 2024Long Context Transfer from Language to Visioncited by 502
  38. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494

50 most cited of 132 All 132 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 239; 2026: 263.

502 papers.

Sort:

By area:

  1. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  2. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  3. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  4. 2025Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoningcited by 217
  5. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  6. 2025SpaceR: Reinforcing MLLMs in Video Spatial Reasoningcited by 152
  7. 2025Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understandingcited by 94
  8. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  9. 2025Scaling RL to Long Videoscited by 83
  10. 2025VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videoscited by 76
  11. 2025STI-Bench: Are MLLMs Ready for Precise Spatial-Temporal World Understanding?cited by 74
  12. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 71
38 more
  1. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 66
  2. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awarenesscited by 61
  3. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  4. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 58
  5. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 57
  6. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 56
  7. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  8. 2025Waver: Wave Your Way to Lifelike Video Generationcited by 52
  9. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  10. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  11. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  12. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 46
  13. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  14. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  15. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  16. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  17. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoningcited by 39
  18. 2025OmniVTLA: Vision-Tactile-Language-Action Models with Semantic-Aligned Tactile Sensingcited by 37
  19. 2025OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipecited by 37
  20. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  21. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  22. 2025Ivy-Fake: A Unified Explainable Framework and Benchmark for Image and Video AIGC Detectioncited by 34
  23. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  24. 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 33
  25. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  26. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 31
  27. 2025MINERVA: Evaluating Complex Video Reasoningcited by 31
  28. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  29. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  30. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  31. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  32. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  33. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  34. 2026VINO: A Unified Visual Generator with Interleaved OmniModal Contextcited by 26
  35. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 25
  36. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  37. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  38. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25

50 most cited of 502 All 502 (JSON)