# DE-RRD: A Knowledge Distillation Framework for Recommender System

arXiv: https://arxiv.org/abs/2012.04357 · HTML: https://mlsanity.com/paper/2012.04357/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Dec 2020.

## Built on by (most cited first, 28 of 28)

- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [Revisiting Self-Distillation](https://mlsanity.com/paper/2206.08491/) (2022)
- [Bootstrapping User and Item Representations for One-Class Collaborative Filtering](https://mlsanity.com/paper/2105.06323/) (2021)
- [DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation](https://mlsanity.com/paper/2206.10848/) (2022)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [Learning Topology-Specific Experts for Molecular Property Prediction](https://mlsanity.com/paper/2302.13693/) (2023)
- [Distillation from Heterogeneous Models for Top-K Recommendation](https://mlsanity.com/paper/2303.01130/) (2023)
- [Taxonomy-guided Semantic Indexing for Academic Paper Search](https://mlsanity.com/paper/2410.19218/) (2024)
- [Unbiased Knowledge Distillation for Recommendation](https://mlsanity.com/paper/2211.14729/) (2022)
- [Privileged Graph Distillation for Cold Start Recommendation](https://mlsanity.com/paper/2105.14975/) (2021)
- [Dual Correction Strategy for Ranking Distillation in Top-N Recommender System](https://mlsanity.com/paper/2109.03459/) (2021)
- [MvFS: Multi-view Feature Selection for Recommender System](https://mlsanity.com/paper/2309.02064/) (2023)
- [Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random](https://mlsanity.com/paper/2403.00817/) (2024)
- [Improving Scientific Document Retrieval with Concept Coverage-based Query Set Generation](https://mlsanity.com/paper/2502.11181/) (2025)
- [Cooperative Retriever and Ranker in Deep Recommenders](https://mlsanity.com/paper/2206.14649/) (2022)
- [RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine](https://mlsanity.com/paper/2410.18097/) (2024)
- [Active Large Language Model-based Knowledge Distillation for Session-based Recommendation](https://mlsanity.com/paper/2502.15685/) (2024)
- [Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filtering](https://mlsanity.com/paper/2202.13140/) (2022)
- [Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search](https://mlsanity.com/paper/2107.07173/) (2021)
- [Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective](https://mlsanity.com/paper/2411.10676/) (2024)
- [Efficient and Stealthy Jailbreak Attacks via Adversarial Prompt Distillation from LLMs to SLMs](https://mlsanity.com/paper/2506.17231/) (2025)
- [Debias the Black-box: A Fair Ranking Framework via Knowledge Distillation](https://mlsanity.com/paper/2208.11628/) (2022)
- [Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems](https://mlsanity.com/paper/2509.20989/) (2025)
- [IRA: Adaptive Interest-aware Representation and Alignment for Personalized Multi-interest Retrieval](https://mlsanity.com/paper/2504.17529/) (2025)
- [BPL: Bias-adaptive Preference Distillation Learning for Recommender System](https://mlsanity.com/paper/2510.16076/) (2025)
- [BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation](https://mlsanity.com/paper/2606.03091/) (2026)

## Abstract (arXiv)

Recent recommender systems have started to employ knowledge distillation, which is a model compression technique distilling knowledge from a cumbersome model (teacher) to a compact model (student), to reduce inference latency while maintaining performance. The state-of-the-art methods have only focused on making the student model accurately imitate the predictions of the teacher model. They have a limitation in that the prediction results incompletely reveal the teacher's knowledge. In this paper, we propose a novel knowledge distillation framework for recommender system, called DE-RRD, which enables the student model to learn from the latent knowledge encoded in the teacher model as well as from the teacher's predictions. Concretely, DE-RRD consists of two methods: 1) Distillation Experts (DE) that directly transfers the latent knowledge from the teacher model. DE exploits "experts" and a novel expert selection strategy for effectively distilling the vast teacher's knowledge to the student with limited capacity. 2) Relaxed Ranking Distillation (RRD) that transfers the knowledge revealed from the teacher's prediction with consideration of the relaxed ranking orders among items. Our extensive experiments show that DE-RRD outperforms the state-of-the-art competitors and achieves comparable or even better performance to that of the teacher model with faster inference time.