# QUEST: Quantized embedding space for transferring knowledge

arXiv: https://arxiv.org/abs/1912.01540 · HTML: https://mlsanity.com/paper/1912.01540/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Dec 2019.

## Built on by (most cited first, 7 of 7)

- [Knowledge Distillation Meets Open-Set Semi-Supervised Learning](https://mlsanity.com/paper/2205.06701/) (2022)
- [MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments](https://mlsanity.com/paper/2307.09361/) (2023)
- [Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2504.13825/) (2025)
- [Few-Shot Object Detection by Knowledge Distillation Using Bag-of-Visual-Words Representations](https://mlsanity.com/paper/2207.12049/) (2022)
- [Knowledge Distillation By Sparse Representation Matching](https://mlsanity.com/paper/2103.17012/) (2021)
- [Knowledge Distillation Layer that Lets the Student Decide](https://mlsanity.com/paper/2309.02843/) (2023)
- [Low-Quality Face Recognition using Center Aligned Representations and Local Margin Constraints](https://mlsanity.com/paper/2609.01014/) (2026)

## Abstract (arXiv)

Knowledge distillation refers to the process of training a compact student network to achieve better accuracy by learning from a high capacity teacher network. Most of the existing knowledge distillation methods direct the student to follow the teacher by matching the teacher's output, feature maps or their distribution. In this work, we propose a novel way to achieve this goal: by distilling the knowledge through a quantized space. According to our method, the teacher's feature maps are quantized to represent the main visual concepts encompassed in the feature maps. The student is then asked to predict the quantized representation, which thus forms the task that the student uses to learn from the teacher. Despite its simplicity, we show that our approach is able to yield results that improve the state of the art on knowledge distillation. To that end, we provide an extensive evaluation across several network architectures and most commonly used benchmark datasets.