# DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval

arXiv: https://arxiv.org/abs/2106.13266 · HTML: https://mlsanity.com/paper/2106.13266/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 37 of its 83 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Billion-scale semi-supervised learning for image classification](https://mlsanity.com/paper/1905.00546/) (2019)
- [Incremental Learning of Object Detectors without Catastrophic Forgetting](https://mlsanity.com/paper/1708.06977/) (2017)
- [Particular object retrieval with integral max-pooling of CNN activations](https://mlsanity.com/paper/1511.05879/) (2015)
- [HashNet: Deep Learning to Hash by Continuation](https://mlsanity.com/paper/1702.00758/) (2017)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Modality Distillation with Multiple Stream Networks for Action Recognition](https://mlsanity.com/paper/1806.07110/) (2018)
- [Central Similarity Quantization for Efficient Image and Video Retrieval](https://mlsanity.com/paper/1908.00347/) (2019)
- [Relation Distillation Networks for Video Object Detection](https://mlsanity.com/paper/1908.09511/) (2019)
- [Evolving Losses for Unsupervised Video Representation Learning](https://mlsanity.com/paper/2002.12177/) (2020)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [D3D: Distilled 3D Networks for Video Action Recognition](https://mlsanity.com/paper/1812.08249/) (2018)
- [Learnable pooling with Context Gating for video classification](https://mlsanity.com/paper/1706.06905/) (2017)
- [Cross-modal knowledge distillation for action recognition](https://mlsanity.com/paper/1910.04641/) (2019)
- [ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning](https://mlsanity.com/paper/1908.07410/) (2019)
- [Graph Distillation for Action Detection with Privileged Modalities](https://mlsanity.com/paper/1712.00108/) (2017)

## Built on by (most cited first, 17 of 17)

- [ILIAS: Instance-Level Image retrieval At Scale](https://mlsanity.com/paper/2502.11748/) (2025)
- [AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrieval](https://mlsanity.com/paper/2408.03282/) (2024)
- [Self-Supervised Video Similarity Learning](https://mlsanity.com/paper/2304.03378/) (2023)
- [The 2023 Video Similarity Dataset and Challenge](https://mlsanity.com/paper/2306.09489/) (2023)
- [Not All Pairs are Equal: Hierarchical Learning for Average-Precision-Oriented Video Retrieval](https://mlsanity.com/paper/2407.15566/) (2024)
- [Evolution of Detection Performance throughout the Online Lifespan of Synthetic Images](https://mlsanity.com/paper/2408.11541/) (2024)
- [The MeVer DeepFake Detection Service: Lessons Learnt from Developing and Deploying in the Wild](https://mlsanity.com/paper/2204.12816/) (2022)
- [Colo-SCRL: Self-Supervised Contrastive Representation Learning for Colonoscopic Video Retrieval](https://mlsanity.com/paper/2303.15671/) (2023)
- [VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression](https://mlsanity.com/paper/2303.08906/) (2023)
- [ConViS-Bench: Estimating Video Similarity Through Semantic Concepts](https://mlsanity.com/paper/2509.19245/) (2025)
- [MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence](https://mlsanity.com/paper/2510.21406/) (2025)
- [TeachCLIP: Multi-Grained Teaching for Efficient Text-to-Video Retrieval](https://mlsanity.com/paper/2308.01217/) (2023)
- [Indexing Multimodal Language Models for Large-scale Image Retrieval](https://mlsanity.com/paper/2604.13268/) (2026)
- [Multi-Focused Video Group Activities Hashing](https://mlsanity.com/paper/2509.00490/) (2025)
- [GPF-Net: Gated Progressive Fusion Learning for Polyp Re-Identification](https://mlsanity.com/paper/2512.21476/) (2025)
- [Efficient Logic Gate Networks for Video Copy Detection](https://mlsanity.com/paper/2604.21694/) (2026)
- [MLT-Dedup: Efficient Large-Scale Online Video Deduplication via Multi-Level Representations and Spatial-Temporal Matching](https://mlsanity.com/paper/2606.12215/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of high performance and computationally efficient content-based video retrieval in large-scale datasets. Current methods typically propose either: (i) fine-grained approaches employing spatio-temporal representations and similarity calculations, achieving high performance at a high computational cost or (ii) coarse-grained approaches representing/indexing videos as global vectors, where the spatio-temporal structure is lost, providing low performance but also having low computational cost. In this work, we propose a Knowledge Distillation framework, called Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: a) Student Networks at different retrieval performance and computational efficiency trade-offs and b) a Selector Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency. We train several students with different architectures and arrive at different trade-offs of performance and efficiency, i.e., speed and storage requirements, including fine-grained students that store/index videos using binary representations. Importantly, the proposed scheme allows Knowledge Distillation in large, unlabelled datasets -- this leads to good students. We evaluate DnS on five public datasets on three different video retrieval tasks and demonstrate a) that our students achieve state-of-the-art performance in several cases and b) that the DnS framework provides an excellent trade-off between retrieval performance, computational speed, and storage space. In specific configurations, the proposed method achieves similar mAP with the teacher but is 20 times faster and requires 240 times less storage space. The collected dataset and implementation are publicly available: https://github.com/mever-team/distill-and-select.