mlsanityData to 7 Oct 2026

UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation

arXiv 2002.06353 ↗First posted 15 Feb 2020cs.CV cs.CL cs.LGcited by 258

Not on the map. Why?

Abstract (arXiv)

With the recent success of the pre-training technique for NLP and image-linguistic tasks, some video-linguistic pre-training works are gradually developed to improve video-text related downstream tasks. However, most of the existing multimodal models are pre-trained for understanding tasks, leading to a pretrain-finetune discrepancy for generation tasks. This paper proposes UniVL: a Unified Video and Language pre-training model for both multimodal understanding and generation. It comprises four components, including two single-modal encoders, a cross encoder, and a decoder with the Transformer backbone. Five objectives, including video-text joint, conditioned masked language model (CMLM), conditioned masked frame model (CMFM), video-text alignment, and language reconstruction, are designed to train each of the components. We further develop two pre-training strategies, stage by stage pre-training (StagedP) and enhanced video representation (EnhancedV), to make the training process of the UniVL more effective. The pre-train is carried out on a sizeable instructional video dataset HowTo100M. Experimental results demonstrate that the UniVL can learn strong video-text representation and achieves state-of-the-art results on five downstream tasks.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 81; 2023: 77; 2024: 53; 2025: 5; 2026: 1.

258 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  5. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  10. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  11. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  12. 2022Egocentric Video-Language Pretrainingcited by 231
38 more
  1. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  2. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  3. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  4. 2022Learning Video Representations from Large Language Modelscited by 204
  5. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  6. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  7. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 162
  8. 2021SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioningcited by 161
  9. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  10. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  11. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  12. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  13. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  14. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  15. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  16. 2023Vision-Language Models as Success Detectorscited by 138
  17. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  18. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  19. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  20. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  21. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  22. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  23. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  24. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  25. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  26. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  27. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  28. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  29. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
  30. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  31. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  32. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  33. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  34. 2022A Simple Multi-Modality Transfer Learning Baseline for Sign Language Translationcited by 80
  35. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  36. 2021Everything at Once -- Multi-modal Fusion Transformer for Video Retrievalcited by 79
  37. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  38. 2023ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Textscited by 78

50 most cited of 258 All 258 (JSON)