# Masked Contrastive Pre-Training for Efficient Video-Text Retrieval

arXiv: https://arxiv.org/abs/2212.00986 · HTML: https://mlsanity.com/paper/2212.00986/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 63 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 30 of 52)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification](https://mlsanity.com/paper/1712.04851/) (2017)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)

## Built on by (most cited first, 11 of 11)

- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [Audio-Visual LLM for Video Understanding](https://mlsanity.com/paper/2312.06720/) (2023)
- [Unified Coarse-to-Fine Alignment for Video-Text Retrieval](https://mlsanity.com/paper/2309.10091/) (2023)
- [Parameter-efficient Tuning of Large-scale Multimodal Foundation Model](https://mlsanity.com/paper/2305.08381/) (2023)
- [Plug-and-Play Regulators for Image-Text Matching](https://mlsanity.com/paper/2303.13371/) (2023)
- [FourierFlow: Frequency-aware Flow Matching for Generative Turbulence Modeling](https://mlsanity.com/paper/2506.00862/) (2025)
- [TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigm](https://mlsanity.com/paper/2409.19865/) (2024)
- [Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrieval](https://mlsanity.com/paper/2305.07910/) (2023)
- [Data Collection-free Masked Video Modeling](https://mlsanity.com/paper/2409.06665/) (2024)
- [Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining](https://mlsanity.com/paper/2603.22953/) (2026)
- [InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding](https://mlsanity.com/paper/2604.08337/) (2026)

## Abstract (arXiv)

We present a simple yet effective end-to-end Video-language Pre-training (VidLP) framework, Masked Contrastive Video-language Pretraining (MAC), for video-text retrieval tasks. Our MAC aims to reduce video representation's spatial and temporal redundancy in the VidLP model by a mask sampling mechanism to improve pre-training efficiency. Comparing conventional temporal sparse sampling, we propose to randomly mask a high ratio of spatial regions and only feed visible regions into the encoder as sparse spatial sampling. Similarly, we adopt the mask sampling technique for text inputs for consistency. Instead of blindly applying the mask-then-prediction paradigm from MAE, we propose a masked-then-alignment paradigm for efficient video-text alignment. The motivation is that video-text retrieval tasks rely on high-level alignment rather than low-level reconstruction, and multimodal alignment with masked modeling encourages the model to learn a robust and general multimodal representation from incomplete and unstable inputs. Coupling these designs enables efficient end-to-end pre-training: reduce FLOPs (60% off), accelerate pre-training (by 3x), and improve performance. Our MAC achieves state-of-the-art results on various video-text retrieval datasets, including MSR-VTT, DiDeMo, and ActivityNet. Our approach is omnivorous to input modalities. With minimal modifications, we achieve competitive results on image-text retrieval tasks.