# Bidirectional Distillation for Top-K Recommender System

arXiv: https://arxiv.org/abs/2106.02870 · HTML: https://mlsanity.com/paper/2106.02870/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2021.

## Built on by (most cited first, 21 of 21)

- [A Comprehensive Survey on Trustworthy Recommender Systems](https://mlsanity.com/paper/2209.10117/) (2022)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [Distillation from Heterogeneous Models for Top-K Recommendation](https://mlsanity.com/paper/2303.01130/) (2023)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [Unbiased Knowledge Distillation for Recommendation](https://mlsanity.com/paper/2211.14729/) (2022)
- [Dual Correction Strategy for Ranking Distillation in Top-N Recommender System](https://mlsanity.com/paper/2109.03459/) (2021)
- [SIGMA: Selective Gated Mamba for Sequential Recommendation](https://mlsanity.com/paper/2408.11451/) (2024)
- [Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation](https://mlsanity.com/paper/2402.02855/) (2024)
- [Doubly Calibrated Estimator for Recommendation on Data Missing Not At Random](https://mlsanity.com/paper/2403.00817/) (2024)
- [Cooperative Retriever and Ranker in Deep Recommenders](https://mlsanity.com/paper/2206.14649/) (2022)
- [Active Large Language Model-based Knowledge Distillation for Session-based Recommendation](https://mlsanity.com/paper/2502.15685/) (2024)
- [Consensus Learning from Heterogeneous Objectives for One-Class Collaborative Filtering](https://mlsanity.com/paper/2202.13140/) (2022)
- [Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective](https://mlsanity.com/paper/2411.10676/) (2024)
- [Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosis](https://mlsanity.com/paper/2412.19654/) (2024)
- [Towards Collaborative Fairness in Federated Learning Under Imbalanced Covariate Shift](https://mlsanity.com/paper/2507.08617/) (2025)
- [Debias the Black-box: A Fair Ranking Framework via Knowledge Distillation](https://mlsanity.com/paper/2208.11628/) (2022)
- [Federated Continual Recommendation](https://mlsanity.com/paper/2508.04792/) (2025)
- [Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems](https://mlsanity.com/paper/2509.20989/) (2025)
- [BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation](https://mlsanity.com/paper/2606.03091/) (2026)
- [GOD: Enhancing Generalization via Deep Grafting for Sequential Recommendation](https://mlsanity.com/paper/2608.16073/) (2026)

## Abstract (arXiv)

Recommender systems (RS) have started to employ knowledge distillation, which is a model compression technique training a compact model (student) with the knowledge transferred from a cumbersome model (teacher). The state-of-the-art methods rely on unidirectional distillation transferring the knowledge only from the teacher to the student, with an underlying assumption that the teacher is always superior to the student. However, we demonstrate that the student performs better than the teacher on a significant proportion of the test set, especially for RS. Based on this observation, we propose Bidirectional Distillation (BD) framework whereby both the teacher and the student collaboratively improve with each other. Specifically, each model is trained with the distillation loss that makes to follow the other's prediction along with its original loss function. For effective bidirectional distillation, we propose rank discrepancy-aware sampling scheme to distill only the informative knowledge that can fully enhance each other. The proposed scheme is designed to effectively cope with a large performance gap between the teacher and the student. Trained in the bidirectional way, it turns out that both the teacher and the student are significantly improved compared to when being trained separately. Our extensive experiments on real-world datasets show that our proposed framework consistently outperforms the state-of-the-art competitors. We also provide analyses for an in-depth understanding of BD and ablation studies to verify the effectiveness of each proposed component.