# Contrastive Masked Autoencoders for Self-Supervised Video Hashing

arXiv: https://arxiv.org/abs/2211.11210 · HTML: https://mlsanity.com/paper/2211.11210/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [Debiased Contrastive Learning](https://mlsanity.com/paper/2007.00224/) (2020)
- [Masked Siamese Networks for Label-Efficient Learning](https://mlsanity.com/paper/2204.07141/) (2022)
- [Context Autoencoder for Self-Supervised Representation Learning](https://mlsanity.com/paper/2202.03026/) (2022)
- [PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers](https://mlsanity.com/paper/2111.12710/) (2021)
- [Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks](https://mlsanity.com/paper/1502.07209/) (2015)
- [Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder](https://mlsanity.com/paper/1802.02305/) (2018)
- [Two-pronged Strategy: Lightweight Augmented Graph Network Hashing for Scalable Image Retrieval](https://mlsanity.com/paper/2108.03914/) (2021)

## Built on by (most cited first, 13 of 13)

- [Unleashing Unlabeled Data: A Paradigm for Cross-View Geo-Localization](https://mlsanity.com/paper/2403.14198/) (2024)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval](https://mlsanity.com/paper/2310.05195/) (2023)
- [Aerial Image Object Detection With Vision Transformer Detector (ViTDet)](https://mlsanity.com/paper/2301.12058/) (2023)
- [CHAIN: Exploring Global-Local Spatio-Temporal Information for Improved Self-Supervised Video Hashing](https://mlsanity.com/paper/2310.18926/) (2023)
- [Efficient Self-Supervised Video Hashing with Selective State Spaces](https://mlsanity.com/paper/2412.14518/) (2024)
- [AutoSSVH: Exploring Automated Frame Sampling for Efficient Self-Supervised Video Hashing](https://mlsanity.com/paper/2504.03587/) (2025)
- [InfoMAE: Pair-Efficient Cross-Modal Alignment for Multimodal Time-Series Sensing Signals](https://mlsanity.com/paper/2504.09707/) (2025)
- [Remote Sensing Scene Classification with Masked Image Modeling (MIM)](https://mlsanity.com/paper/2302.14256/) (2023)
- [Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling](https://mlsanity.com/paper/2405.02941/) (2024)
- [Invertible Residual Rescaling Models](https://mlsanity.com/paper/2405.02945/) (2024)
- [Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval](https://mlsanity.com/paper/2408.05711/) (2024)
- [Multi-Focused Video Group Activities Hashing](https://mlsanity.com/paper/2509.00490/) (2025)

## Abstract (arXiv)

Self-Supervised Video Hashing (SSVH) models learn to generate short binary representations for videos without ground-truth supervision, facilitating large-scale video retrieval efficiency and attracting increasing research attention. The success of SSVH lies in the understanding of video content and the ability to capture the semantic relation among unlabeled videos. Typically, state-of-the-art SSVH methods consider these two points in a two-stage training pipeline, where they firstly train an auxiliary network by instance-wise mask-and-predict tasks and secondly train a hashing model to preserve the pseudo-neighborhood structure transferred from the auxiliary network. This consecutive training strategy is inflexible and also unnecessary. In this paper, we propose a simple yet effective one-stage SSVH method called ConMH, which incorporates video semantic information and video similarity relationship understanding in a single stage. To capture video semantic information for better hashing learning, we adopt an encoder-decoder structure to reconstruct the video from its temporal-masked frames. Particularly, we find that a higher masking ratio helps video understanding. Besides, we fully exploit the similarity relationship between videos by maximizing agreement between two augmented views of a video, which contributes to more discriminative and robust hash codes. Extensive experiments on three large-scale video datasets (i.e., FCVID, ActivityNet and YFCC) indicate that ConMH achieves state-of-the-art results. Code is available at https://github.com/huangmozhi9527/ConMH.