mlsanityData to 14 Oct 2026

Unmasked Teacher: Towards Training-Efficient Video Foundation Models

arXiv 2303.16058 ↗First posted 28 Mar 2023cs.CVcited by 199

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video Foundation Models (VFMs) have received limited exploration due to high computational costs and data scarcity. Previous VFMs rely on Image Foundation Models (IFMs), which face challenges in transferring to the video domain. Although VideoMAE has trained a robust ViT from limited data, its low-level reconstruction poses convergence difficulties and conflicts with high-level cross-modal alignment. This paper proposes a training-efficient method for temporal-sensitive VFMs that integrates the benefits of existing methods. To increase data efficiency, we mask out most of the low-semantics video tokens, but selectively align the unmasked tokens with IFM, which serves as the UnMasked Teacher (UMT). By providing semantic guidance, our method enables faster convergence and multimodal friendliness. With a progressive pre-training framework, our model can handle various tasks including scene-related, temporal-related, and complex video-language understanding. Using only public sources for pre-training in 6 days on 32 A100 GPUs, our scratch-built ViT-L/16 achieves state-of-the-art performances on various video tasks. The code and models will be released at https://github.com/OpenGVLab/unmasked_teacher.

Key facts. Cited by 199 papers in the map (data through 14 Oct 2026). Builds on 92 of its 111 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

92 of 111 references.

References · 92

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  7. 2016Layer Normalizationcited by 5,669
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2017The Kinetics Human Action Video Datasetcited by 1,909
  8. 2018SlowFast Networks for Video Recognitioncited by 1,867
  9. 2017Decoupled Weight Decay Regularizationcited by 1,831
  10. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  11. 2021ViViT: A Video Vision Transformercited by 1,620
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  13. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  14. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  15. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  19. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  20. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  21. 2021Video Swin Transformercited by 1,091
  22. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  23. 2021Scaling Vision Transformerscited by 999
  24. 2016Deep Networks with Stochastic Depthcited by 979
  25. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  26. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  27. 2021Multiscale Vision Transformerscited by 897
  28. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  29. 2017Dense-Captioning Events in Videoscited by 793
  30. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  31. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  32. 2017Localizing Moments in Video with Natural Languagecited by 663
  33. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  34. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  35. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  36. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  37. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 519
  38. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 25; 2024: 76; 2025: 76; 2026: 22.

199 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  6. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  7. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  8. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  9. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 328
  10. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  11. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
  12. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
38 more
  1. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 220
  2. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 169
  3. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 147
  4. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  5. 2026V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learningcited by 117
  6. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 110
  7. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  8. 2024TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuningcited by 106
  9. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  10. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 105
  11. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 98
  12. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  13. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 89
  14. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 87
  15. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 75
  16. 2024TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Abilitycited by 70
  17. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  18. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  19. 2025Towards Understanding Camera Motions in Any Videocited by 64
  20. 2025Video-Bench: Human-Aligned Video Generation Benchmarkcited by 57
  21. 2025LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learningcited by 55
  22. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  23. 2024Gramian Multimodal Representation Learning and Alignmentcited by 54
  24. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  25. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 45
  26. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 43
  27. 2023BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuningcited by 41
  28. 2024vid-TLDR: Training Free Token merging for Light-weight Video Transformercited by 41
  29. 2024VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Modelscited by 41
  30. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  31. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  32. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  33. 2024Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentationcited by 39
  34. 2024A Survey on Generative AI and LLM for Video Generation, Understanding, and Streamingcited by 37
  35. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 35
  36. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  37. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  38. 2024GAIA: Rethinking Action Quality Assessment for AI-Generated Videoscited by 30

50 most cited of 199 All 199 (JSON)