# Does Knowledge Distillation Matter for Large Language Model based Bundle Generation?

arXiv: https://arxiv.org/abs/2504.17220 · HTML: https://mlsanity.com/paper/2504.17220/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 41 of its 74 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 30 of 41)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering](https://mlsanity.com/paper/1602.01585/) (2016)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation](https://mlsanity.com/paper/2305.00447/) (2023)
- [Uncovering ChatGPT's Capabilities in Recommender Systems](https://mlsanity.com/paper/2305.02182/) (2023)
- [Large Language Models as Zero-Shot Conversational Recommenders](https://mlsanity.com/paper/2308.10053/) (2023)
- [Is ChatGPT Fair for Recommendation? Evaluating Fairness in Large Language Model Recommendation](https://mlsanity.com/paper/2305.07609/) (2023)
- [Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation](https://mlsanity.com/paper/2404.05970/) (2024)
- [Leveraging Large Language Models for Sequential Recommendation](https://mlsanity.com/paper/2309.09261/) (2023)
- [ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation](https://mlsanity.com/paper/2308.11131/) (2023)
- [Generative Recommendation: Towards Next-generation Recommender Paradigm](https://mlsanity.com/paper/2304.03516/) (2023)
- [POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion](https://mlsanity.com/paper/1905.01866/) (2019)
- [Enhancing Recommender Systems with Large Language Model Reasoning Graphs](https://mlsanity.com/paper/2308.10835/) (2023)
- [Exploring Large Language Model for Graph Data Understanding in Online Job Recommendations](https://mlsanity.com/paper/2307.05722/) (2023)
- [Large Language Models for Intent-Driven Session Recommendations](https://mlsanity.com/paper/2312.07552/) (2023)
- [Diffusion Models for Generative Outfit Recommendation](https://mlsanity.com/paper/2402.17279/) (2024)
- [CrossCBR: Cross-view Contrastive Learning for Bundle Recommendation](https://mlsanity.com/paper/2206.00242/) (2022)
- [Can Small Language Models be Good Reasoners for Sequential Recommendation?](https://mlsanity.com/paper/2403.04260/) (2024)
- [Bundle Recommendation with Graph Convolutional Networks](https://mlsanity.com/paper/2005.03475/) (2020)
- [Multi-view Intent Disentangle Graph Networks for Bundle Recommendation](https://mlsanity.com/paper/2202.11425/) (2022)
- [RDRec: Rationale Distillation for LLM-based Recommendation](https://mlsanity.com/paper/2405.10587/) (2024)
- [Personalized Bundle List Recommendation](https://mlsanity.com/paper/1904.01933/) (2019)
- [NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application](https://mlsanity.com/paper/2102.04887/) (2021)
- [Tiny-NewsRec: Effective and Efficient PLM-based News Recommendation](https://mlsanity.com/paper/2112.00944/) (2021)
- [Personalized Bundle Recommendation in Online Games](https://mlsanity.com/paper/2104.05307/) (2021)
- [Distillation Matters: Empowering Sequential Recommenders to Match the Performance of Large Language Model](https://mlsanity.com/paper/2405.00338/) (2024)
- [Knowledge Prompt-tuning for Sequential Recommendation](https://mlsanity.com/paper/2308.08459/) (2023)

## Built on by (most cited first, 1 of 1)

- [Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation](https://mlsanity.com/paper/2508.17250/) (2025)

## Abstract (arXiv)

LLMs are increasingly explored for bundle generation, thanks to their reasoning capabilities and knowledge. However, deploying large-scale LLMs introduces significant efficiency challenges, primarily high computational costs during fine-tuning and inference due to their massive parameterization. Knowledge distillation (KD) offers a promising solution, transferring expertise from large teacher models to compact student models. This study systematically investigates knowledge distillation approaches for bundle generation, aiming to minimize computational demands while preserving performance. We explore three critical research questions: (1) how does the format of KD impact bundle generation performance? (2) to what extent does the quantity of distilled knowledge influence performance? and (3) how do different ways of utilizing the distilled knowledge affect performance? We propose a comprehensive KD framework that (i) progressively extracts knowledge (patterns, rules, deep thoughts); (ii) captures varying quantities of distilled knowledge through different strategies; and (iii) exploits complementary LLM adaptation techniques (in-context learning, supervised fine-tuning, combination) to leverage distilled knowledge in small student models for domain-specific adaptation and enhanced efficiency. Extensive experiments provide valuable insights into how knowledge format, quantity, and utilization methodologies collectively shape LLM-based bundle generation performance, exhibiting KD's significant potential for more efficient yet effective LLM-based bundle generation.