mlsanityData to 14 Oct 2026

VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs

arXiv 2406.07476 ↗First posted 11 Jun 2024cs.CV cs.CLcited by 771

Not on the map. Why?

Cited by 316 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this paper, we present the VideoLLaMA 2, a set of Video Large Language Models (Video-LLMs) designed to enhance spatial-temporal modeling and audio understanding in video and audio-oriented tasks. Building upon its predecessor, VideoLLaMA 2 incorporates a tailor-made Spatial-Temporal Convolution (STC) connector, which effectively captures the intricate spatial and temporal dynamics of video data. Additionally, we integrate an Audio Branch into the model through joint training, thereby enriching the multimodal understanding capabilities of the model by seamlessly incorporating audio cues. Comprehensive evaluations on multiple-choice video question answering (MC-VQA), open-ended video question answering (OE-VQA), and video captioning (VC) tasks demonstrate that VideoLLaMA 2 consistently achieves competitive results among open-source models and even gets close to some proprietary models on several benchmarks. Furthermore, VideoLLaMA 2 exhibits reasonable improvements in audio-only and audio-video question-answering (AQA & OE-AVQA) benchmarks over existing models. These advancements underline VideoLLaMA 2's superior performance in multimodal comprehension, setting a new standard for intelligent video analysis systems. All models are public to facilitate further research.

Key facts. Cited by 771 papers in the map (data through 14 Oct 2026). Builds on 97 of its 126 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

97 of 126 references.

References · 97

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2017The Kinetics Human Action Video Datasetcited by 1,909
  5. 2024Mixtral of Expertscited by 1,823
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  8. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  12. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  13. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  14. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  15. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  16. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  17. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  18. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  19. 2020Designing Network Design Spacescited by 960
  20. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  21. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  22. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  23. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  24. 2024Yi: Open Foundation Models by 01.AIcited by 719
  25. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  26. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  27. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  28. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  29. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  30. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
  31. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  32. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  33. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  34. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  35. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  36. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  37. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
  38. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461

50 most cited of 97 All 97 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 119; 2025: 375; 2026: 277.

771 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  5. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  6. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 366
  7. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 356
  8. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  9. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  10. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  11. 2025Kimi-Audio Technical Reportcited by 222
  12. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
38 more
  1. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  2. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  3. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  5. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 181
  6. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  7. 2025WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMscited by 155
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  9. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  10. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 137
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  12. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 127
  13. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 125
  14. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  15. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  16. 2025Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalitiescited by 110
  17. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  18. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  19. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  20. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 97
  21. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 96
  22. 2025StreamingVLM: Real-Time Understanding for Infinite Video Streamscited by 95
  23. 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 93
  24. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  25. 2025Scaling RL to Long Videoscited by 84
  26. 2024Frame-Voyager: Learning to Query Frames for Video Large Language Modelscited by 83
  27. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  28. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 80
  29. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  30. 2024CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Drivingcited by 77
  31. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  32. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 73
  33. 2024VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMcited by 72
  34. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 70
  35. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 70
  36. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  37. 2024VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understandingcited by 68
  38. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 65

50 most cited of 771 All 771 (JSON)