mlsanityData to 13 Oct 2026

InternVideo2: Scaling Foundation Models for Multimodal Video Understanding

arXiv 2403.15377 ↗First posted 22 Mar 2024cs.CVcited by 311

Not on the map. Why?

Cited by 79 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce InternVideo2, a new family of video foundation models (ViFM) that achieve the state-of-the-art results in video recognition, video-text tasks, and video-centric dialogue. Our core design is a progressive training approach that unifies the masked video modeling, crossmodal contrastive learning, and next token prediction, scaling up the video encoder size to 6B parameters. At the data level, we prioritize spatiotemporal consistency by semantically segmenting videos and generating video-audio-speech captions. This improves the alignment between video and text. Through extensive experiments, we validate our designs and demonstrate superior performance on over 60 video and audio tasks. Notably, our model outperforms others on various video-related dialogue and long video understanding benchmarks, highlighting its ability to reason and comprehend longer contexts. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2/.

Key facts. Cited by 311 papers in the map (data through 13 Oct 2026). Builds on 105 of its 126 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

105 of 126 references.

References · 105

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  5. 2023GPT-4 Technical Reportcited by 18,278
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  10. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,111
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  12. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,857
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  6. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,004
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,264
  9. 2023Mistral 7Bcited by 3,123
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,546
  13. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  14. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  15. 2018SlowFast Networks for Video Recognitioncited by 1,864
  16. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,535
  17. 2019Root Mean Square Layer Normalizationcited by 1,449
  18. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,355
  19. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,308
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
  21. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,176
  22. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,164
  23. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,150
  24. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  25. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,116
  26. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,093
  27. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,092
  28. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,052
  29. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  30. 2016Bag of Tricks for Efficient Text Classificationcited by 872
  31. 2023VideoChat: Chat-Centric Video Understandingcited by 854
  32. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 792
  33. 2017Dense-Captioning Events in Videoscited by 790
  34. 2024Genie: Generative Interactive Environmentscited by 721
  35. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 693
  36. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 672
  37. 2017Localizing Moments in Video with Natural Languagecited by 661
  38. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 650

50 most cited of 105 All 105 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 25; 2025: 156; 2026: 130.

311 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,586
  2. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 769
  3. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 319
  4. 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 250
  5. 2025World Simulation with Video Foundation Models for Physical AIcited by 190
  6. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  7. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 108
  8. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  9. 2025WISA: World Simulator Assistant for Physics-Aware Text-to-Video Generationcited by 89
  10. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 83
  11. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 76
  12. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 73
38 more
  1. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  2. 2024PruneVid: Visual Token Pruning for Efficient Video Large Language Modelscited by 65
  3. 2025Towards Understanding Camera Motions in Any Videocited by 62
  4. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  5. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 58
  6. 2025VideoRAG: Retrieval-Augmented Generation over Video Corpuscited by 57
  7. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 54
  8. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 52
  9. 2024Foundation Models for Video Understanding: A Surveycited by 46
  10. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 45
  11. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  12. 2025Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsificationcited by 36
  13. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 34
  14. 2025EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Modelscited by 32
  15. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 30
  16. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 27
  17. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  18. 2024ViLLa: Video Reasoning Segmentation with Large Language Modelcited by 23
  19. 2024From Specific-MLLMs to Omni-MLLMs: A Survey on MLLMs Aligned with Multi-modalitiescited by 22
  20. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 22
  21. 2025Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrievalcited by 20
  22. 2025UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularitiescited by 20
  23. 2025Better Tokens for Better 3D: Advancing Vision-Language Modeling in 3D Medical Imagingcited by 20
  24. 2024Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Modelcited by 19
  25. 2025A Survey on Video Temporal Grounding with Multimodal Large Language Modelcited by 19
  26. 2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 18
  27. 2024Chrono: A Simple Blueprint for Representing Time in MLLMscited by 17
  28. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  29. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoTcited by 16
  30. 2025TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMscited by 15
  31. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  32. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  33. 2025SAIL-VL2 Technical Reportcited by 14
  34. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  35. 2025MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrievalcited by 13
  36. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  37. 2025ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Modelscited by 13
  38. 2025Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fieldscited by 13

50 most cited of 311 All 311 (JSON)