# Unbiased Knowledge Distillation for Recommendation

arXiv: https://arxiv.org/abs/2211.14729 · HTML: https://mlsanity.com/paper/2211.14729/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 27 Nov 2022.

## Placed in

- Knowledge distillation / [Distillation for recommender systems](https://mlsanity.com/distill/distillation-for-recommender-systems/)

## Builds on (most cited first, 25 of 25)

- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Counterfactual VQA: A Cause-Effect Look at Language Bias](https://mlsanity.com/paper/2006.04315/) (2020)
- [Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effect](https://mlsanity.com/paper/2009.12991/) (2020)
- [Recommendations as Treatments: Debiasing Learning and Evaluation](https://mlsanity.com/paper/1602.05352/) (2016)
- [Bias and Debias in Recommender System: A Survey and Future Directions](https://mlsanity.com/paper/2010.03240/) (2020)
- [Distilling Causal Effect of Data in Class-Incremental Learning](https://mlsanity.com/paper/2103.01737/) (2021)
- [Causal Intervention for Leveraging Popularity Bias in Recommendation](https://mlsanity.com/paper/2105.06067/) (2021)
- [Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System](https://mlsanity.com/paper/2010.15363/) (2020)
- [Causal Embeddings for Recommendation](https://mlsanity.com/paper/1706.07639/) (2017)
- [AutoDebias: Learning to Debias for Recommendation](https://mlsanity.com/paper/2105.04170/) (2021)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [DE-RRD: A Knowledge Distillation Framework for Recommender System](https://mlsanity.com/paper/2012.04357/) (2020)
- [Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method](https://mlsanity.com/paper/2010.11797/) (2020)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [Cross-Task Knowledge Distillation in Multi-Task Recommendation](https://mlsanity.com/paper/2202.09852/) (2022)
- [Bidirectional Distillation for Top-K Recommender System](https://mlsanity.com/paper/2106.02870/) (2021)
- [Top-N Recommendation with Counterfactual User Preference Simulation](https://mlsanity.com/paper/2109.02444/) (2021)
- [On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation](https://mlsanity.com/paper/2204.11091/) (2022)
- [Ensemble Knowledge Distillation for CTR Prediction](https://mlsanity.com/paper/2011.04106/) (2020)
- [UKD: Debiasing Conversion Rate Estimation via Uncertainty-regularized Knowledge Distillation](https://mlsanity.com/paper/2201.08024/) (2022)
- [Collaborative Distillation for Top-N Recommendation](https://mlsanity.com/paper/1911.05276/) (2019)
- [Crank up the volume: preference bias amplification in collaborative recommendation](https://mlsanity.com/paper/1909.06362/) (2019)
- [Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation](https://mlsanity.com/paper/1912.08422/) (2019)

## Built on by (most cited first, 11 of 11)

- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [External Large Foundation Model: How to Efficiently Serve Trillions of Parameters for Online Ads Recommendation](https://mlsanity.com/paper/2502.17494/) (2025)
- [Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation](https://mlsanity.com/paper/2402.02855/) (2024)
- [LightGNN: Simple Graph Neural Network for Recommendation](https://mlsanity.com/paper/2501.03228/) (2025)
- [Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective](https://mlsanity.com/paper/2411.10676/) (2024)
- [Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems](https://mlsanity.com/paper/2509.20989/) (2025)
- [DIIT: A Domain-Invariant Information Transfer Method for Industrial Cross-Domain Recommendation](https://mlsanity.com/paper/2410.10835/) (2024)
- [BPL: Bias-adaptive Preference Distillation Learning for Recommender System](https://mlsanity.com/paper/2510.16076/) (2025)
- [MBD: A Model-Based Debiasing Framework Across User, Content, and Model Dimensions](https://mlsanity.com/paper/2603.14422/) (2026)
- [LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation](https://mlsanity.com/paper/2605.29280/) (2026)
- [BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation](https://mlsanity.com/paper/2606.03091/) (2026)

## Abstract (arXiv)

As a promising solution for model compression, knowledge distillation (KD) has been applied in recommender systems (RS) to reduce inference latency. Traditional solutions first train a full teacher model from the training data, and then transfer its knowledge (ie textitsoft labels) to supervise the learning of a compact student model. However, we find such a standard distillation paradigm would incur serious bias issue -- popular items are more heavily recommended after the distillation. This effect prevents the student model from making accurate and fair recommendations, decreasing the effectiveness of RS. In this work, we identify the origin of the bias in KD -- it roots in the biased soft labels from the teacher, and is further propagated and intensified during the distillation. To rectify this, we propose a new KD method with a stratified distillation strategy. It first partitions items into multiple groups according to their popularity, and then extracts the ranking knowledge within each group to supervise the learning of the student. Our method is simple and teacher-agnostic -- it works on distillation stage without affecting the training of the teacher model. We conduct extensive theoretical and empirical studies to validate the effectiveness of our proposal. We release our code at: https://github.com/chengang95/UnKD.