mlsanityData to 14 Oct 2026

VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling

arXiv 2111.12681 ↗First posted 24 Nov 2021cs.CVcited by 188

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. Recent studies try to mitigate this disconnection via end-to-end training. To make it computationally feasible, prior works tend to "imagify" video inputs, i.e., a handful of sparsely sampled frames are fed into a 2D CNN, followed by a simple mean-pooling or concatenation to obtain the overall video representations. Although achieving promising results, such simple approaches may lose temporal information that is essential for performing downstream VidL tasks. In this work, we present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs. Further, unlike previous studies that found pre-training tasks on video inputs (e.g., masked frame modeling) not very effective, we design a new pre-training task, Masked Visual-token Modeling (MVM), for better video modeling. Specifically, the original video frame patches are "tokenized" into discrete visual tokens, and the goal is to recover the original visual tokens based on the masked patches. Comprehensive analysis demonstrates the effectiveness of both explicit temporal modeling via video transformer and MVM. As a result, VIOLET achieves new state-of-the-art performance on 5 video question answering tasks and 4 text-to-video retrieval tasks.

Key facts. Cited by 188 papers in the map (data through 14 Oct 2026). Builds on 76 of its 88 references that are in the map. First posted on arXiv 24 Nov 2021.

Builds on

76 of 88 references.

References · 76

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2017Neural Discrete Representation Learningcited by 4,712
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
38 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  4. 2017The Kinetics Human Action Video Datasetcited by 1,909
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  11. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  13. 2021Video Swin Transformercited by 1,091
  14. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  15. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  16. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  17. 2017Dense-Captioning Events in Videoscited by 793
  18. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  19. 2017Localizing Moments in Video with Natural Languagecited by 663
  20. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  21. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 619
  22. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  23. 2017TALL: Temporal Activity Localization via Language Querycited by 584
  24. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  25. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  26. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  27. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  28. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  29. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  30. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  31. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  32. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  33. 2020Multi-modal Transformer for Video Retrievalcited by 395
  34. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  35. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  36. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
  37. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  38. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 38; 2023: 81; 2024: 43; 2025: 16; 2026: 9.

188 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  2. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  3. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  4. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  7. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  8. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  9. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  10. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  12. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
38 more
  1. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  2. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  3. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  4. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  5. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  6. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  7. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
  8. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  9. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  10. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  11. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  12. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  13. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  14. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 125
  15. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  16. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  17. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 114
  18. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  19. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  20. 2023In-Context Learning Unlocked for Diffusion Modelscited by 104
  21. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  22. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  23. 2022Video Question Answering: Datasets, Algorithms and Challengescited by 84
  24. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 83
  25. 2023Audio-Visual LLM for Video Understandingcited by 77
  26. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  27. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  28. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  29. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  30. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  31. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 71
  32. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 70
  33. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 67
  34. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  35. 2022Video Graph Transformer for Video Question Answeringcited by 63
  36. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  37. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  38. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55

50 most cited of 188 All 188 (JSON)