# Explainable LLM-driven Multi-dimensional Distillation for E-Commerce Relevance Learning arXiv: https://arxiv.org/abs/2411.13045 ยท HTML: https://mlsanity.com/paper/2411.13045/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. - Builds on 30 of its 55 references that are in the map. - First posted on arXiv 20 Nov 2024. ## Placed in - Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/) ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023) - [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022) - [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023) - [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020) - [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023) - [PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization](https://mlsanity.com/paper/2306.05087/) (2023) - [OpenChat: Advancing Open-source Language Models with Mixed-Quality Data](https://mlsanity.com/paper/2309.11235/) (2023) - [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022) - [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016) - [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023) - [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019) - [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022) - [AugGPT: Leveraging ChatGPT for Text Data Augmentation](https://mlsanity.com/paper/2302.13007/) (2023) - [Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction](https://mlsanity.com/paper/2303.04132/) (2023) - [Convolutional Neural Network Architectures for Matching Natural Language Sentences](https://mlsanity.com/paper/1503.03244/) (2015) - [Text Matching as Image Recognition](https://mlsanity.com/paper/1602.06359/) (2016) - [Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty](https://mlsanity.com/paper/2308.02019/) (2023) - [Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval](https://mlsanity.com/paper/1502.06922/) (2015) - [Match-SRNN: Modeling the Recursive Matching Structure with Spatial RNN](https://mlsanity.com/paper/1604.04378/) (2016) - [QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation](https://mlsanity.com/paper/2210.15718/) (2022) - [Semantic Modelling with Long-Short-Term Memory for Information Retrieval](https://mlsanity.com/paper/1412.6629/) (2014) - [Targeted Data Generation: Finding and Fixing Model Weaknesses](https://mlsanity.com/paper/2305.17804/) (2023) - [LinkNER: Linking Local Named Entity Recognition Models to Large Language Models using Uncertainty](https://mlsanity.com/paper/2402.10573/) (2024) - [Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce](https://mlsanity.com/paper/2407.09395/) (2024) ## Built on by (most cited first, 4 of 4) - [Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search](https://mlsanity.com/paper/2512.00968/) (2025) - [Generating Query-Relevant Document Summaries via Reinforcement Learning](https://mlsanity.com/paper/2508.08404/) (2025) - [LORE: A Large Generative Model for Search Relevance](https://mlsanity.com/paper/2512.03025/) (2025) - [From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction](https://mlsanity.com/paper/2610.16569/) (2026) ## Abstract (arXiv) Effective query-item relevance modeling is pivotal for enhancing user experience and safeguarding user satisfaction in e-commerce search systems. Recently, benefiting from the vast inherent knowledge, Large Language Model (LLM) approach demonstrates strong performance and long-tail generalization ability compared with previous neural-based specialized relevance learning methods. Though promising, current LLM-based methods encounter the following inadequacies in practice: First, the massive parameters and computational demands make it difficult to be deployed online. Second, distilling LLM models to online models is a feasible direction, but the LLM relevance modeling is a black box, and its rich intrinsic knowledge is difficult to extract and apply online. To improve the interpretability of LLM and boost the performance of online relevance models via LLM, we propose an Explainable LLM-driven Multi-dimensional Distillation framework for e-commerce relevance learning, which comprises two core components: (1) An Explainable LLM for relevance modeling (ELLM-rele), which decomposes the relevance learning into intermediate steps and models relevance learning as a Chain-of-Thought (CoT) reasoning, thereby enhancing both interpretability and performance of LLM. (2) A Multi-dimensional Knowledge Distillation (MKD) architecture that transfers the knowledge of ELLM-rele to current deployable interaction-based and representation-based student models from both the relevance score distribution and CoT reasoning aspects. Through distilling the probabilistic and CoT reasoning knowledge, MKD improves both the semantic interaction and long-tail generalization abilities of student models. Extensive offline evaluations and online experiments on Taobao search ad scene demonstrate that our proposed framework significantly enhances e-commerce relevance learning performance and user experience.