mlsanityData to 14 Oct 2026

VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding

arXiv 2109.14084 ↗First posted 28 Sep 2021cs.CV cs.CLcited by 602

Not on the map. Why?

Cited by 95 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present VideoCLIP, a contrastive approach to pre-train a unified model for zero-shot video and text understanding, without using any labels on downstream tasks. VideoCLIP trains a transformer for video and text by contrasting temporally overlapping positive video-text pairs with hard negatives from nearest neighbor retrieval. Our experiments on a diverse series of downstream tasks, including sequence-level text-video retrieval, VideoQA, token-level action localization, and action segmentation reveal state-of-the-art performance, surpassing prior work, and in some cases even outperforming supervised approaches. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

Key facts. Cited by 602 papers in the map (data through 14 Oct 2026). Builds on 58 of its 68 references that are in the map. First posted on arXiv 28 Sep 2021.

Builds on

58 of 68 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2018Deep contextualized word representationscited by 2,290
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2018SlowFast Networks for Video Recognitioncited by 1,867
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  9. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  12. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  13. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  14. 2017Localizing Moments in Video with Natural Languagecited by 663
  15. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  16. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  17. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  18. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  19. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  20. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  21. 2020Multi-modal Transformer for Video Retrievalcited by 395
  22. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  23. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  24. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  25. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  26. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  27. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  28. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  29. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  30. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  31. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 198
  32. 2020Support-set bottlenecks for video-text representation learningcited by 170
  33. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  34. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  35. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  36. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  37. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  38. 2020Pre-training via Paraphrasingcited by 92

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 7; 2022: 92; 2023: 143; 2024: 156; 2025: 117; 2026: 87.

602 papers.

Sort:

By area:

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  5. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  7. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 531
  8. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  11. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  12. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
38 more
  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  2. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  3. 2022Multimodal Learning with Transformers: A Surveycited by 400
  4. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  5. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  6. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  7. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  8. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  9. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  10. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  11. 2022Learning Video Representations from Large Language Modelscited by 205
  12. 2023Text-To-4D Dynamic Scene Generationcited by 205
  13. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  14. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  15. 2023Multimodal Large Language Models: A Surveycited by 195
  16. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  17. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 188
  18. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  19. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 187
  20. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  21. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 181
  22. 2022Revisiting the "Video" in Video-Language Understandingcited by 169
  23. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  24. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  25. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 141
  26. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  27. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 134
  28. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  29. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  30. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  31. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 126
  32. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  33. 2023VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detectioncited by 123
  34. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  35. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  36. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  37. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  38. 2024TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Modelscited by 97

50 most cited of 602 All 602 (JSON)