# Bridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems

arXiv: https://arxiv.org/abs/2408.14678 · HTML: https://mlsanity.com/paper/2408.14678/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation.
- Builds on 4 of its 10 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Placed in

- Knowledge distillation / [Distillation for recommender systems](https://mlsanity.com/distill/distillation-for-recommender-systems/)

## Builds on (most cited first, 4 of 4)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Improving Training Stability for Multitask Ranking Models in Recommender Systems](https://mlsanity.com/paper/2302.09178/) (2023)

## Abstract (arXiv)

Knowledge Distillation (KD) is a powerful approach for compressing a large model into a smaller, more efficient model, particularly beneficial for latency-sensitive applications like recommender systems. However, current KD research predominantly focuses on Computer Vision (CV) and NLP tasks, overlooking unique data characteristics and challenges inherent to recommender systems. This paper addresses these overlooked challenges, specifically: (1) mitigating data distribution shifts between teacher and student models, (2) efficiently identifying optimal teacher configurations within time and budgetary constraints, and (3) enabling computationally efficient and rapid sharing of teacher labels to support multiple students. We present a robust KD system developed and rigorously evaluated on multiple large-scale personalized video recommendation systems within Google. Our live experiment results demonstrate significant improvements in student model performance while ensuring consistent and reliable generation of high quality teacher labels from a continuous data stream of data.