mlsanityData to 7 Oct 2026

An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling

arXiv 2209.01540 ↗First posted 4 Sep 2022cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

Masked visual modeling (MVM) has been recently proven effective for visual pre-training. While similar reconstructive objectives on video inputs (e.g., masked frame modeling) have been explored in video-language (VidL) pre-training, previous studies fail to find a truly effective MVM strategy that can largely benefit the downstream performance. In this work, we systematically examine the potential of MVM in the context of VidL learning. Specifically, we base our study on a fully end-to-end VIdeO-LanguagE Transformer (VIOLET), where the supervision from MVM training can be backpropagated to the video pixel space. In total, eight different reconstructive targets of MVM are explored, from low-level pixel values and oriented gradients to high-level depth maps, optical flow, discrete visual tokens, and latent visual features. We conduct comprehensive experiments and provide insights into the factors leading to effective MVM training, resulting in an enhanced model VIOLETv2. Empirically, we show VIOLETv2 pre-trained with MVM objective achieves notable improvements on 13 VidL benchmarks, ranging from video question answering, video captioning, to text-to-video retrieval.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 76 of its 93 references that are in the map. First posted on arXiv 4 Sep 2022.

Builds on

76 of 93 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  5. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2018SlowFast Networks for Video Recognitioncited by 1,862
  8. 2021Vision Transformers for Dense Predictioncited by 1,789
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  14. 2021Video Swin Transformercited by 1,090
  15. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  16. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  17. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  18. 2017Dense-Captioning Events in Videoscited by 789
  19. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  20. 2017Localizing Moments in Video with Natural Languagecited by 660
  21. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  22. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  23. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  24. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  25. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  26. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  27. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  28. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  29. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  30. 2020Multi-modal Transformer for Video Retrievalcited by 395
  31. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  32. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  33. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 320
  34. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  35. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  36. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  37. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  38. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 28; 2024: 19; 2025: 18; 2026: 5.

74 papers.

Sort:

By area:

  1. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  2. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  4. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 223
  5. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  6. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  7. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  8. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 133
  9. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  10. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  11. 2025Perception-Aware Policy Optimization for Multimodal Reasoningcited by 95
  12. 2024Video ReCap: Recursive Captioning of Hour-Long Videoscited by 88
38 more
  1. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  2. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  3. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  4. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  5. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  6. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  7. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  8. 2024Language Repository for Long Video Understandingcited by 55
  9. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  10. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  11. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  12. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  13. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  14. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  15. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  16. 2024Masked Image Modeling: A Surveycited by 28
  17. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  18. 2023MaMMUT: A Simple Architecture for Joint Learning for MultiModal Taskscited by 24
  19. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  20. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Modelscited by 21
  21. 2023Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocolscited by 17
  22. 2025HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understandingcited by 17
  23. 2023Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalitiescited by 15
  24. 2022SMAUG: Sparse Masked Autoencoder for Efficient Video-Language Pre-trainingcited by 14
  25. 2022Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learningcited by 14
  26. 2023Global and Local Semantic Completion Learning for Vision-Language Pre-trainingcited by 12
  27. 2025Cross-modal Causal Relation Alignment for Video Question Groundingcited by 12
  28. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  29. 2023COSA: Concatenated Sample Pretrained Vision-Language Foundation Modelcited by 11
  30. 2024Long Story Short: Story-level Video Understanding from 20K Short Filmscited by 11
  31. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11
  32. 2023CAD -- Contextual Multi-modal Alignment for Dynamic AVQAcited by 10
  33. 2024Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answeringcited by 9
  34. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
  35. 2024MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learningcited by 8
  36. 2025VideoExpert: Augmented LLM for Temporal-Sensitive Video Understandingcited by 8
  37. 2023Multi Sentence Description of Complex Manipulation Action Videoscited by 6
  38. 2023Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Promptscited by 5

50 most cited of 74 All 74 (JSON)