mlsanityData to 7 Oct 2026

Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

arXiv 2111.10337 ↗First posted 19 Nov 2021cs.CVcited by 226

Not on the map. Why?

Cited by 38 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study joint video and language (VL) pre-training to enable cross-modality learning and benefit plentiful downstream VL tasks. Existing works either extract low-quality video features or learn limited text embedding, while neglecting that high-resolution videos and diversified semantics can significantly improve cross-modality learning. In this paper, we propose a novel High-resolution and Diversified VIdeo-LAnguage pre-training model (HD-VILA) for many visual tasks. In particular, we collect a large dataset with two distinct properties: 1) the first high-resolution dataset including 371.5k hours of 720p videos, and 2) the most diversified dataset covering 15 popular YouTube categories. To enable VL pre-training, we jointly optimize the HD-VILA model by a hybrid Transformer that learns rich spatiotemporal features, and a multimodal Transformer that enforces interactions of the learned video features with diversified texts. Our pre-training model achieves new state-of-the-art results in 10 VL understanding tasks and 2 more novel text-to-visual generation tasks. For example, we outperform SOTA models with relative increases of 40.4% R@1 in zero-shot MSR-VTT text-to-video retrieval task and 55.4% in high-resolution dataset LSMDC. The learned VL embedding is also effective in generating visually pleasing and semantically relevant results in text-to-visual editing and super-resolution tasks.

Key facts. Cited by 226 papers in the map (data through 7 Oct 2026). Builds on 59 of its 81 references that are in the map. First posted on arXiv 19 Nov 2021.

Builds on

59 of 81 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  7. 2014Going Deeper with Convolutionscited by 4,231
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  11. 2018SlowFast Networks for Video Recognitioncited by 1,862
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
38 more
  1. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  2. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  3. 2018Datasheets for Datasetscited by 1,371
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2017Dense-Captioning Events in Videoscited by 789
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2017Localizing Moments in Video with Natural Languagecited by 660
  13. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  14. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  15. 2021Designing an Encoder for StyleGAN Image Manipulationcited by 542
  16. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  17. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  18. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  19. 2020Multi-modal Transformer for Video Retrievalcited by 395
  20. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  21. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  22. 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
  23. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  24. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  25. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  26. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  27. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  28. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  29. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  30. 2016Movie Descriptioncited by 180
  31. 2020Support-set bottlenecks for video-text representation learningcited by 170
  32. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  33. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  34. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  35. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  36. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  37. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  38. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 53; 2024: 77; 2025: 63; 2026: 22.

226 papers.

Sort:

By area:

  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2023ModelScope Text-to-Video Technical Reportcited by 651
  4. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  5. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  6. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  8. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  9. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  10. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 366
  11. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  12. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  2. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  5. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  6. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  7. 2023A Survey on Video Diffusion Modelscited by 203
  8. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  9. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  10. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  11. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  12. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  13. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  14. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  15. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  16. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  17. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  18. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  19. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  20. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  21. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  22. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  23. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  24. 2024VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understandingcited by 113
  25. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  26. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  27. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  28. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  29. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  30. 2025Controllable Video Generation: A Surveycited by 93
  31. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  32. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  33. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  34. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  35. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  36. 2024Vript: A Video Is Worth Thousands of Wordscited by 71
  37. 2024Vlogger: Make Your Dream A Vlogcited by 70
  38. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68

50 most cited of 226 All 226 (JSON)