# Active Large Language Model-based Knowledge Distillation for Session-based Recommendation

arXiv: https://arxiv.org/abs/2502.15685 · HTML: https://mlsanity.com/paper/2502.15685/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Distillation for recommender systems, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 38 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Placed in

- Knowledge distillation / [Distillation for recommender systems](https://mlsanity.com/distill/distillation-for-recommender-systems/)

## Builds on (most cited first, 24 of 24)

- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)](https://mlsanity.com/paper/2203.13366/) (2022)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Large Language Models are Zero-Shot Rankers for Recommender Systems](https://mlsanity.com/paper/2305.08845/) (2023)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [Representation Learning with Large Language Models for Recommendation](https://mlsanity.com/paper/2310.15950/) (2023)
- [Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction](https://mlsanity.com/paper/2305.06474/) (2023)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [Efficiently Leveraging Multi-level User Intent for Session-based Recommendation via Atten-Mixer Network](https://mlsanity.com/paper/2206.12781/) (2022)
- [Large Language Models for Intent-Driven Session Recommendations](https://mlsanity.com/paper/2312.07552/) (2023)
- [DE-RRD: A Knowledge Distillation Framework for Recommender System](https://mlsanity.com/paper/2012.04357/) (2020)
- [Integrating Large Language Models into Recommendation via Mutual Augmentation and Adaptive Aggregation](https://mlsanity.com/paper/2401.13870/) (2024)
- [Can Small Language Models be Good Reasoners for Sequential Recommendation?](https://mlsanity.com/paper/2403.04260/) (2024)
- [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021)
- [Bidirectional Distillation for Top-K Recommender System](https://mlsanity.com/paper/2106.02870/) (2021)
- [On-Device Next-Item Recommendation with Self-Supervised Knowledge Distillation](https://mlsanity.com/paper/2204.11091/) (2022)
- [DaisyRec 2.0: Benchmarking Recommendation for Rigorous Evaluation](https://mlsanity.com/paper/2206.10848/) (2022)
- [Collaborative Distillation for Top-N Recommendation](https://mlsanity.com/paper/1911.05276/) (2019)
- [Distillation from Heterogeneous Models for Top-K Recommendation](https://mlsanity.com/paper/2303.01130/) (2023)
- [Distilling Structured Knowledge into Embeddings for Explainable and Accurate Recommendation](https://mlsanity.com/paper/1912.08422/) (2019)
- [Privileged Graph Distillation for Cold Start Recommendation](https://mlsanity.com/paper/2105.14975/) (2021)
- [Dual Correction Strategy for Ranking Distillation in Top-N Recommender System](https://mlsanity.com/paper/2109.03459/) (2021)
- [Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation](https://mlsanity.com/paper/2402.02855/) (2024)
- [Knowledge Distillation Approaches for Accurate and Efficient Recommender System](https://mlsanity.com/paper/2407.13952/) (2024)

## Built on by (most cited first, 6 of 6)

- [Bidirectional Knowledge Distillation for Enhancing Sequential Recommendation with Large Language Models](https://mlsanity.com/paper/2505.18120/) (2025)
- [Reinforcement Speculative Decoding for Fast Ranking](https://mlsanity.com/paper/2505.20316/) (2025)
- [Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasets](https://mlsanity.com/paper/2510.06240/) (2025)
- [A Comparative Study of Traditional Machine Learning, Deep Learning, and Large Language Models for Mental Health Forecasting using Smartphone Sensing Data](https://mlsanity.com/paper/2601.03603/) (2026)
- [Causal Direct Preference Optimization for Distributionally Robust Generative Recommendation](https://mlsanity.com/paper/2603.22335/) (2026)
- [How Vulnerable Are Edge LLMs?](https://mlsanity.com/paper/2603.23822/) (2026)

## Abstract (arXiv)

Large language models (LLMs) provide a promising way for accurate session-based recommendation (SBR), but they demand substantial computational time and memory. Knowledge distillation (KD)-based methods can alleviate these issues by transferring the knowledge to a small student, which trains a student based on the predictions of a cumbersome teacher. However, these methods encounter difficulties for textitLLM-based KD in SBR. 1) It is expensive to make LLMs predict for all instances in KD. 2) LLMs may make ineffective predictions for some instances in KD, e.g., incorrect predictions for hard instances or similar predictions as existing recommenders for easy instances. In this paper, we propose an active LLM-based KD method in SBR, contributing to sustainable AI. To efficiently distill knowledge from LLMs with limited cost, we propose to extract a small proportion of instances predicted by LLMs. Meanwhile, for a more effective distillation, we propose an active learning strategy to extract instances that are as effective as possible for KD from a theoretical view. Specifically, we first formulate gains based on potential effects (e.g., effective, similar, and incorrect predictions by LLMs) and difficulties (e.g., easy or hard to fit) of instances for KD. Then, we propose to maximize the minimal gains of distillation to find the optimal selection policy for active learning, which can largely avoid extracting ineffective instances in KD. Experiments on real-world datasets show that our method significantly outperforms state-of-the-art methods for SBR.