mlsanityData to 14 Oct 2026

Exploring Feature-based Knowledge Distillation for Recommender System: A Frequency Perspective

arXiv 2411.10676 ↗First posted 16 Nov 2024cs.IR cs.AI cs.LGcited by 3

On the map: Knowledge distillation → Distillation for recommender systems

Abstract (arXiv)

In this paper, we analyze the feature-based knowledge distillation for recommendation from the frequency perspective. By defining knowledge as different frequency components of the features, we theoretically demonstrate that regular feature-based knowledge distillation is equivalent to equally minimizing losses on all knowledge and further analyze how this equal loss weight allocation method leads to important knowledge being overlooked. In light of this, we propose to emphasize important knowledge by redistributing knowledge weights. Furthermore, we propose FreqD, a lightweight knowledge reweighting method, to avoid the computational cost of calculating losses on each knowledge. Extensive experiments demonstrate that FreqD consistently and significantly outperforms state-of-the-art knowledge distillation methods for recommender systems. Our code is available at https://github.com/woriazzc/KDs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation. Builds on 30 of its 48 references that are in the map. First posted on arXiv 16 Nov 2024.

Builds on

30 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Unbiased Knowledge Distillation for Recommendation27 Nov
  2. 2021Dual Correction Strategy for Ranking Distillation in Top-N Recommender System8 Sep
  3. 2021Topology Distillation for Recommender System16 Jun

Other references · 27

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018How Powerful are Graph Neural Networks?cited by 4,393
  3. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  4. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
  7. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  8. 2019Relational Knowledge Distillationcited by 889
  9. 2019Neural Graph Collaborative Filteringcited by 661
  10. 2018Variational Autoencoders for Collaborative Filteringcited by 310
  11. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  12. 2022Knowledge Distillation from A Stronger Teachercited by 208
15 more
  1. 2019A Survey on Session-based Recommender Systemscited by 97
  2. 2021SimpleX: A Simple and Strong Baseline for Collaborative Filteringcited by 65
  3. 2018Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender Systemcited by 60
  4. 2022Towards Representation Alignment and Uniformity in Collaborative Filteringcited by 57
  5. 2021A Survey of Machine Learning for Computer Architecture and Systemscited by 53
  6. 2021How Powerful is Graph Convolution for Recommendation?cited by 50
  7. 2018Learning Student Networks via Feature Embeddingcited by 45
  8. 2020A Survey on Trajectory Data Management, Analytics, and Learningcited by 41
  9. 2022Blurring-Sharpening Process Models for Collaborative Filteringcited by 35
  10. 2020DE-RRD: A Knowledge Distillation Framework for Recommender Systemcited by 28
  11. 2023Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Frameworkcited by 24
  12. 2021Bidirectional Distillation for Top-K Recommender Systemcited by 21
  13. 2019Collaborative Distillation for Top-N Recommendationcited by 13
  14. 2022Less is More: Reweighting Important Spectral Graph Features for Recommendationcited by 10
  15. 2024Continual Collaborative Distillation for Recommender Systemcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Rejuvenating Cross-Entropy Loss in Knowledge Distillation for Recommender Systemscited by 1
  2. 2025Beyond Semantic Understanding: Preserving Collaborative Frequency Components in LLM-based Recommendationcited by 0
  3. 2026BAHSD: Bridging the Long-tail Gap via Adaptive Distillation in Black-box Sequential Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Knowledge Distillation Approaches for Accurate and Efficient Recommender System
  2. 2023Distillation from Heterogeneous Models for Top-K Recommendation
  3. 2024Bridging the Gap: Unpacking the Hidden Challenges in Knowledge Distillation for Online Ranking Systems
  4. 2024Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model
  5. 2025Bidirectional Knowledge Distillation for Enhancing Sequential Recommendation with Large Language Models
  6. 2024Pre-trained Language Model and Knowledge Distillation for Lightweight Sequential Recommendation
  7. 2022Cross-Task Knowledge Distillation in Multi-Task Recommendation
  8. 2024Enhancing Scalability in Recommender Systems through Lottery Ticket Hypothesis and Knowledge Distillation-based Neural Network Pruning