# Bit-mask Robust Contrastive Knowledge Distillation for Unsupervised Semantic Hashing

arXiv: https://arxiv.org/abs/2403.06071 · HTML: https://mlsanity.com/paper/2403.06071/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 65 references that are in the map.
- First posted on arXiv 10 Mar 2024.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [A Survey on Learning to Hash](https://mlsanity.com/paper/1606.00185/) (2016)
- [Central Similarity Quantization for Efficient Image and Video Retrieval](https://mlsanity.com/paper/1908.00347/) (2019)
- [A Survey on Deep Hashing Methods](https://mlsanity.com/paper/2003.03369/) (2020)
- [Self-supervised Product Quantization for Deep Unsupervised Image Retrieval](https://mlsanity.com/paper/2109.02244/) (2021)
- [DistillHash: Unsupervised Deep Hashing by Distilling Data Pairs](https://mlsanity.com/paper/1905.03465/) (2019)
- [Auto-Encoding Twin-Bottleneck Hashing](https://mlsanity.com/paper/2002.11930/) (2020)
- [Unsupervised Hashing with Contrastive Information Bottleneck](https://mlsanity.com/paper/2105.06138/) (2021)
- [Binary Generative Adversarial Networks for Image Retrieval](https://mlsanity.com/paper/1708.04150/) (2017)
- [Deep Unsupervised Image Hashing by Maximizing Bit Entropy](https://mlsanity.com/paper/2012.12334/) (2020)
- [Stochastic Generative Hashing](https://mlsanity.com/paper/1701.02815/) (2017)
- [Privileged Graph Distillation for Cold Start Recommendation](https://mlsanity.com/paper/2105.14975/) (2021)
- [Deep Hash Distillation for Image Retrieval](https://mlsanity.com/paper/2112.08816/) (2021)
- [Unsupervised Multi-Index Semantic Hashing](https://mlsanity.com/paper/2103.14460/) (2021)
- [A General Two-Step Approach to Learning-Based Hashing](https://mlsanity.com/paper/1309.1853/) (2013)

## Abstract (arXiv)

Unsupervised semantic hashing has emerged as an indispensable technique for fast image search, which aims to convert images into binary hash codes without relying on labels. Recent advancements in the field demonstrate that employing large-scale backbones (e.g., ViT) in unsupervised semantic hashing models can yield substantial improvements. However, the inference delay has become increasingly difficult to overlook. Knowledge distillation provides a means for practical model compression to alleviate this delay. Nevertheless, the prevailing knowledge distillation approaches are not explicitly designed for semantic hashing. They ignore the unique search paradigm of semantic hashing, the inherent necessities of the distillation process, and the property of hash codes. In this paper, we propose an innovative Bit-mask Robust Contrastive knowledge Distillation (BRCD) method, specifically devised for the distillation of semantic hashing models. To ensure the effectiveness of two kinds of search paradigms in the context of semantic hashing, BRCD first aligns the semantic spaces between the teacher and student models through a contrastive knowledge distillation objective. Additionally, to eliminate noisy augmentations and ensure robust optimization, a cluster-based method within the knowledge distillation process is introduced. Furthermore, through a bit-level analysis, we uncover the presence of redundancy bits resulting from the bit independence property. To mitigate these effects, we introduce a bit mask mechanism in our knowledge distillation objective. Finally, extensive experiments not only showcase the noteworthy performance of our BRCD method in comparison to other knowledge distillation methods but also substantiate the generality of our methods across diverse semantic hashing models and backbones. The code for BRCD is available at https://github.com/hly1998/BRCD.