mlsanityData to 14 Oct 2026

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

arXiv 2105.09996 ↗First posted 20 May 2021cs.CV cs.CLcited by 102

Not on the map. Why?

Abstract (arXiv)

We present a simplified, task-agnostic multi-modal pre-training approach that can accept either video or text input, or both for a variety of end tasks. Existing pre-training are task-specific by adopting either a single cross-modal encoder that requires both modalities, limiting their use for retrieval-style end tasks or more complex multitask learning with two unimodal encoders, limiting early cross-modal fusion. We instead introduce new pretraining masking schemes that better mix across modalities (e.g. by forcing masks for text to predict the closest video embeddings) while also maintaining separability (e.g. unimodal predictions are sometimes required, without using all the input). Experimental results show strong performance across a wider range of tasks than any previous methods, often outperforming task-specific pre-training. Code is made available at https://github.com/pytorch/fairseq/tree/main/examples/MMPT.

Key facts. Cited by 102 papers in the map (data through 14 Oct 2026). Builds on 38 of its 41 references that are in the map. First posted on arXiv 20 May 2021.

Builds on

38 of 41 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
26 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  2. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  3. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  4. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  5. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  6. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  7. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  8. 2020Multi-modal Transformer for Video Retrievalcited by 395
  9. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  10. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  11. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  12. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  13. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  14. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  15. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  16. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  17. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  18. 2020Support-set bottlenecks for video-text representation learningcited by 170
  19. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  20. 2020AVLnet: Learning Audio-Visual Language Representations from Instructional Videoscited by 111
  21. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  22. 2018Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignmentcited by 74
  23. 2018NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learningcited by 67
  24. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Modelscited by 41
  25. 2019Learning Video Representations using Contrastive Bidirectional Transformercited by 39
  26. 2020Video Understanding as Machine Translationcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 27; 2023: 34; 2024: 22; 2025: 9; 2026: 6.

102 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  3. 2022Multimodal Learning with Transformers: A Surveycited by 400
  4. 2022Egocentric Video-Language Pretrainingcited by 232
  5. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  6. 2023Video Understanding with Large Language Models: A Surveycited by 206
  7. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 193
  8. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  9. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  10. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  11. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 89
  12. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
38 more
  1. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  2. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  3. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  4. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  5. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  6. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 70
  7. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  8. 2023UnLoc: A Unified Framework for Video Localization Taskscited by 66
  9. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  10. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 54
  11. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  12. 2023NaQ: Leveraging Narrations as Queries to Supervise Episodic Memorycited by 46
  13. 2023Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructionscited by 46
  14. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  15. 2022Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Videocited by 43
  16. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  17. 2023ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Modelcited by 39
  18. 2022Temporal Sentence Grounding in Videos: A Survey and Future Directionscited by 37
  19. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  20. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  21. 2023Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrievalcited by 35
  22. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  23. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 28
  24. 2023GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generationcited by 28
  25. 2021Survey: Transformer based Video-Language Pre-trainingcited by 27
  26. 2023Differentially Private Attention Computationcited by 25
  27. 2022Large-Scale Retrieval for Reinforcement Learningcited by 24
  28. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  29. 2023Tem-adapter: Adapting Image-Text Pretraining for Video Question Answercited by 22
  30. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  31. 2022End-to-End Modeling via Information Tree for One-Shot Natural Language Spatial Video Groundingcited by 19
  32. 2023Accommodating Audio Modality in CLIP for Multimodal Processingcited by 18
  33. 2023SpotEM: Efficient Video Search for Episodic Memorycited by 17
  34. 2023RGNet: A Unified Clip Retrieval and Grounding Network for Long Videoscited by 15
  35. 2022Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videoscited by 14
  36. 2025AutoOcc: Automatic Open-Ended Semantic Occupancy Annotation via Vision-Language Guided Gaussian Splattingcited by 14
  37. 2022Hierarchical3D Adapters for Long Video-to-text Summarizationcited by 12
  38. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs
  2. 2022All in One: Exploring Unified Video-Language Pre-training
  3. 2022A Survey of Vision-Language Pre-Trained Models
  4. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
  5. 2022VLP: A Survey on Vision-Language Pre-training
  6. 2023Audio-Visual LLM for Video Understanding
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding
  8. 2023Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks