# Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective

arXiv: https://arxiv.org/abs/2411.10676 · HTML: https://mlsanity.com/paper/2411.10676/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 16 Nov 2024.

## Placed in

- Knowledge distillation / [Distillation for recommender systems](https://mlsanity.com/distill/distillation-for-recommender-systems/)

## Builds on (most cited first, 30 of 30)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Neural Graph Collaborative Filtering](https://mlsanity.com/paper/1905.08108/) (2019)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Knowledge Distillation from A Stronger Teacher](https://mlsanity.com/paper/2205.10536/) (2022)
- [A Survey on Session-based Recommender Systems](https://mlsanity.com/paper/1902.04864/) (2019)
- [SimpleX: A Simple and Strong Baseline for Collaborative Filtering](https://mlsanity.com/paper/2109.12613/) (2021)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [Towards Representation Alignment and Uniformity in Collaborative Filtering](https://mlsanity.com/paper/2206.12811/) (2022)
- [A Survey of Machine Learning for Computer Architecture and Systems](https://mlsanity.com/paper/2102.07952/) (2021)
- [How Powerful is Graph Convolution for Recommendation?](https://mlsanity.com/paper/2108.07567/) (2021)
- [Learning Student Networks via Feature Embedding](https://mlsanity.com/paper/1812.06597/) (2018)
- [A Survey on Trajectory Data Management, Analytics, and Learning](https://mlsanity.com/paper/2003.11547/) (2020)
- [Blurring-Sharpening Process Models for Collaborative Filtering](https://mlsanity.com/paper/2211.09324/) (2022)
- [DE-RRD: A Knowledge Distillation Framework for Recommender System](https://mlsanity.com/paper/2012.04357/) (2020)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework](https://mlsanity.com/paper/2305.10758/) (2023)
- [Bidirectional Distillation for Top-K Recommender System](https://mlsanity.com/paper/2106.02870/) (2021)
- [Collaborative Distillation for Top-N Recommendation](https://mlsanity.com/paper/1911.05276/) (2019)
- [Unbiased Knowledge Distillation for Recommendation](https://mlsanity.com/paper/2211.14729/) (2022)
- [Dual Correction Strategy for Ranking Distillation in Top-N Recommender System](https://mlsanity.com/paper/2109.03459/) (2021)
- [Less is More: Reweighting Important Spectral Graph Features for Recommendation](https://mlsanity.com/paper/2204.11346/) (2022)
- [Continual Collaborative Distillation for Recommender System](https://mlsanity.com/paper/2405.19046/) (2024)

## Built on by (most cited first, 3 of 3)

- [Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systems](https://mlsanity.com/paper/2509.20989/) (2025)
- [Beyond Semantic Understanding: Preserving Collaborative Frequency Components in LLM-based Recommendation](https://mlsanity.com/paper/2508.10312/) (2025)
- [BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendation](https://mlsanity.com/paper/2606.03091/) (2026)

## Abstract (arXiv)

In this paper, we analyze the feature-based knowledge distillation for recommendation from the frequency perspective. By defining knowledge as different frequency components of the features, we theoretically demonstrate that regular feature-based knowledge distillation is equivalent to equally minimizing losses on all knowledge and further analyze how this equal loss weight allocation method leads to important knowledge being overlooked. In light of this, we propose to emphasize important knowledge by redistributing knowledge weights. Furthermore, we propose FreqD, a lightweight knowledge reweighting method, to avoid the computational cost of calculating losses on each knowledge. Extensive experiments demonstrate that FreqD consistently and significantly outperforms state-of-the-art knowledge distillation methods for recommender systems. Our code is available at https://github.com/woriazzc/KDs.