# Context Unaware Knowledge Distillation for Image Retrieval

arXiv: https://arxiv.org/abs/2207.09070 · HTML: https://mlsanity.com/paper/2207.09070/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 19 Jul 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [Central Similarity Quantization for Efficient Image and Video Retrieval](https://mlsanity.com/paper/1908.00347/) (2019)
- [A Decade Survey of Content Based Image Retrieval using Deep Learning](https://mlsanity.com/paper/2012.00641/) (2020)
- [Deep Triplet Quantization](https://mlsanity.com/paper/1902.00153/) (2019)
- [Vision Transformer Hashing for Image Retrieval](https://mlsanity.com/paper/2109.12564/) (2021)
- [Frequency Disentangled Residual Network](https://mlsanity.com/paper/2109.12556/) (2021)

## Built on by (most cited first, 1 of 1)

- [Face to Cartoon Incremental Super-Resolution using Knowledge Distillation](https://mlsanity.com/paper/2401.15366/) (2024)

## Abstract (arXiv)

Existing data-dependent hashing methods use large backbone networks with millions of parameters and are computationally complex. Existing knowledge distillation methods use logits and other features of the deep (teacher) model and as knowledge for the compact (student) model, which requires the teacher's network to be fine-tuned on the context in parallel with the student model on the context. Training teacher on the target context requires more time and computational resources. In this paper, we propose context unaware knowledge distillation that uses the knowledge of the teacher model without fine-tuning it on the target context. We also propose a new efficient student model architecture for knowledge distillation. The proposed approach follows a two-step process. The first step involves pre-training the student model with the help of context unaware knowledge distillation from the teacher model. The second step involves fine-tuning the student model on the context of image retrieval. In order to show the efficacy of the proposed approach, we compare the retrieval results, no. of parameters and no. of operations of the student models with the teacher models under different retrieval frameworks, including deep cauchy hashing (DCH) and central similarity quantization (CSQ). The experimental results confirm that the proposed approach provides a promising trade-off between the retrieval results and efficiency. The code used in this paper is released publicly at urlhttps://github.com/satoru2001/CUKDFIR.