mlsanityData to 14 Oct 2026

InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation

arXiv 2307.06942 ↗First posted 13 Jul 2023cs.CVcited by 497

Not on the map. Why?

Cited by 107 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces InternVid, a large-scale video-centric multimodal dataset that enables learning powerful and transferable video-text representations for multimodal understanding and generation. The InternVid dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M video clips accompanied by detailed descriptions of total 4.1B words. Our core contribution is to develop a scalable approach to autonomously build a high-quality video-text dataset with large language models (LLM), thereby showcasing its efficacy in learning video-language representation at scale. Specifically, we utilize a multi-scale approach to generate video-related descriptions. Furthermore, we introduce ViCLIP, a video-text representation learning model based on ViT-L. Learned on InternVid via contrastive learning, this model demonstrates leading zero-shot action recognition and competitive video retrieval performance. Beyond basic video understanding tasks like recognition and retrieval, our dataset and model have broad applications. They are particularly beneficial for generating interleaved video-text data for learning a video-centric dialogue system, advancing video-to-text and text-to-video generation research. These proposed resources provide a tool for researchers and practitioners interested in multimodal video understanding and generation.

Key facts. Cited by 497 papers in the map (data through 14 Oct 2026). Builds on 73 of its 84 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

73 of 84 references.

References · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
38 more
  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  2. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  3. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  5. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  11. 2017Dense-Captioning Events in Videoscited by 793
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  13. 2017Localizing Moments in Video with Natural Languagecited by 663
  14. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  15. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  16. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  17. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  18. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  19. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  20. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 528
  21. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  22. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  23. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 331
  24. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  25. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  26. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  27. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  28. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  29. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  30. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  31. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  32. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  33. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  34. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  35. 2015A Dataset for Movie Descriptioncited by 195
  36. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  37. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  38. 2016Movie Descriptioncited by 181

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 21; 2024: 162; 2025: 182; 2026: 132.

497 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  5. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  6. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  8. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  10. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  11. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  12. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
38 more
  1. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 255
  2. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 251
  3. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 233
  4. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  5. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 228
  6. 2025TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Modelscited by 213
  7. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  8. 2023A Survey on Video Diffusion Modelscited by 206
  9. 2023Video Understanding with Large Language Models: A Surveycited by 206
  10. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  11. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 185
  12. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  13. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 147
  14. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 146
  15. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  16. 2025Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Modelscited by 143
  17. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  18. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 135
  19. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 132
  20. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 131
  21. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  22. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistencycited by 117
  23. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  24. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  25. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  26. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  27. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  28. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  29. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  30. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 100
  31. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 91
  32. 2024FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attentioncited by 89
  33. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  34. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 87
  35. 2023Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videoscited by 86
  36. 2024HawkEye: Training Video-Text LLMs for Grounding Text in Videoscited by 86
  37. 2025Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compressioncited by 85
  38. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81

50 most cited of 497 All 497 (JSON)