# LREF: A Novel LLM-based Relevance Framework for E-commerce

arXiv: https://arxiv.org/abs/2503.09223 · HTML: https://mlsanity.com/paper/2503.09223/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Sequence-to-Sequence Learning as Beam-Search Optimization](https://mlsanity.com/paper/1606.02960/) (2016)
- [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024)
- [Determinants of LLM-assisted Decision-Making](https://mlsanity.com/paper/2402.17385/) (2024)
- [Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challenges](https://mlsanity.com/paper/2403.00784/) (2024)
- [A Multi-Granularity Matching Attention Network for Query Intent Classification in E-commerce Retrieval](https://mlsanity.com/paper/2303.15870/) (2023)
- [Large Language Models for Relevance Judgment in Product Search](https://mlsanity.com/paper/2406.00247/) (2024)
- [Knowledge Distillation based Contextual Relevance Matching for E-commerce Product Search](https://mlsanity.com/paper/2210.01701/) (2022)
- [An Interpretable Ensemble of Graph and Language Models for Improving Search Relevance in E-Commerce](https://mlsanity.com/paper/2403.00923/) (2024)
- [A Semi-supervised Multi-channel Graph Convolutional Network for Query Classification in E-commerce](https://mlsanity.com/paper/2408.01928/) (2024)
- [Beyond Keywords and Relevance: A Personalized Ad Retrieval Framework in E-Commerce Sponsored Search](https://mlsanity.com/paper/1712.10110/) (2017)

## Built on by (most cited first, 8 of 8)

- [Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search](https://mlsanity.com/paper/2512.00968/) (2025)
- [TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance](https://mlsanity.com/paper/2510.08048/) (2025)
- [Towards Reliable Evaluation of Large Language Models for Multilingual and Multimodal E-Commerce Applications](https://mlsanity.com/paper/2510.20632/) (2025)
- [LORE: A Large Generative Model for Search Relevance](https://mlsanity.com/paper/2512.03025/) (2025)
- [SERM: Self-Evolving Relevance Model with Agent-Driven Learning from Massive Query Streams](https://mlsanity.com/paper/2601.09515/) (2026)
- [SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search](https://mlsanity.com/paper/2609.29803/) (2026)
- [RAG-Match: Retrieval-Augmented Knowledge Injection and Hierarchical Reasoning for Calibrated Semantic Relevance](https://mlsanity.com/paper/2605.25486/) (2026)
- [From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction](https://mlsanity.com/paper/2610.16569/) (2026)

## Abstract (arXiv)

Query and product relevance prediction is a critical component for ensuring a smooth user experience in e-commerce search. Traditional studies mainly focus on BERT-based models to assess the semantic relevance between queries and products. However, the discriminative paradigm and limited knowledge capacity of these approaches restrict their ability to comprehend the relevance between queries and products fully. With the rapid advancement of Large Language Models (LLMs), recent research has begun to explore their application to industrial search systems, as LLMs provide extensive world knowledge and flexible optimization for reasoning processes. Nonetheless, directly leveraging LLMs for relevance prediction tasks introduces new challenges, including a high demand for data quality, the necessity for meticulous optimization of reasoning processes, and an optimistic bias that can result in over-recall. To overcome the above problems, this paper proposes a novel framework called the LLM-based RElevance Framework (LREF) aimed at enhancing e-commerce search relevance. The framework comprises three main stages: supervised fine-tuning (SFT) with Data Selection, Multiple Chain of Thought (Multi-CoT) tuning, and Direct Preference Optimization (DPO) for de-biasing. We evaluate the performance of the framework through a series of offline experiments on large-scale real-world datasets, as well as online A/B testing. The results indicate significant improvements in both offline and online metrics. Ultimately, the model was deployed in a well-known e-commerce application, yielding substantial commercial benefits.