# Learning a Product Relevance Model from Click-Through Data in E-Commerce

arXiv: https://arxiv.org/abs/2102.07098 · HTML: https://mlsanity.com/paper/2102.07098/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2021.

## Built on by (most cited first, 22 of 22)

- [Ranking Manipulation for Conversational Search Engines](https://mlsanity.com/paper/2406.03589/) (2024)
- [MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2301.12646/) (2023)
- [Large Language Models for Relevance Judgment in Product Search](https://mlsanity.com/paper/2406.00247/) (2024)
- [Multi-Objective Personalized Product Retrieval in Taobao Search](https://mlsanity.com/paper/2210.04170/) (2022)
- [Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting](https://mlsanity.com/paper/2408.09439/) (2024)
- [Commonsense Knowledge Salience Evaluation with a Benchmark Dataset in E-commerce](https://mlsanity.com/paper/2205.10843/) (2022)
- [Beyond Semantics: Learning a Behavior Augmented Relevance Model with Self-supervised Learning](https://mlsanity.com/paper/2308.05379/) (2023)
- [Deep Bag-of-Words Model: An Efficient and Interpretable Relevance Architecture for Chinese E-Commerce](https://mlsanity.com/paper/2407.09395/) (2024)
- [Semantic Retrieval at Walmart](https://mlsanity.com/paper/2412.04637/) (2024)
- [Delving into E-Commerce Product Retrieval with Vision-Language Pre-training](https://mlsanity.com/paper/2304.04377/) (2023)
- [SPM: Structured Pretraining and Matching Architectures for Relevance Modeling in Meituan Search](https://mlsanity.com/paper/2308.07711/) (2023)
- [CWRCzech: 100M Query-Document Czech Click Dataset and Its Application to Web Relevance Ranking](https://mlsanity.com/paper/2405.20994/) (2024)
- [SUMMA: A Multimodal Large Language Model for Advertisement Summarization](https://mlsanity.com/paper/2508.20582/) (2025)
- [Deep Intention-Aware Network for Click-Through Rate Prediction](https://mlsanity.com/paper/2211.08650/) (2022)
- [Improving Text Matching in E-Commerce Search with A Rationalizable, Intervenable and Fast Entity-Based Relevance Model](https://mlsanity.com/paper/2307.00370/) (2023)
- [Towards Better Query Classification with Multi-Expert Knowledge Condensation in JD Ads Search](https://mlsanity.com/paper/2308.01098/) (2023)
- [Improving Topic Relevance Model by Mix-structured Summarization and LLM-based Data Augmentation](https://mlsanity.com/paper/2404.02616/) (2024)
- [Adaptively Learning to Select-Rank in Online Platforms](https://mlsanity.com/paper/2406.05017/) (2024)
- [PRECTR: A Synergistic Framework for Integrating Personalized Search Relevance Matching and CTR Prediction](https://mlsanity.com/paper/2503.18395/) (2025)
- [Personalized Product Search Ranking: A Multi-Task Learning Approach with Tabular and Non-Tabular Data](https://mlsanity.com/paper/2508.09636/) (2025)
- [Mine and Refine: Optimizing Graded Relevance in E-commerce Semantic Search Retrieval](https://mlsanity.com/paper/2602.17654/) (2026)
- [Semantic Retrieval for Product Search in E-Commerce](https://mlsanity.com/paper/2606.01504/) (2026)

## Abstract (arXiv)

The search engine plays a fundamental role in online e-commerce systems, to help users find the products they want from the massive product collections. Relevance is an essential requirement for e-commerce search, since showing products that do not match search query intent will degrade user experience. With the existence of vocabulary gap between user language of queries and seller language of products, measuring semantic relevance is necessary and neural networks are engaged to address this task. However, semantic relevance is different from click-through rate prediction in that no direct training signal is available. Most previous attempts learn relevance models from user click-through data that are cheap and abundant. Unfortunately, click behavior is noisy and misleading, which is affected by not only relevance but also factors including price, image and attractive titles. Therefore, it is challenging but valuable to learn relevance models from click-through data. In this paper, we propose a new relevance learning framework that concentrates on how to train a relevance model from the weak supervision of click-through data. Different from previous efforts that treat samples as either relevant or irrelevant, we construct more fine-grained samples for training. We propose a novel way to consider samples of different relevance confidence, and come up with a new training objective to learn a robust relevance model with desirable score distribution. The proposed model is evaluated on offline annotated data and online A/B testing, and it achieves both promising performance and high computational efficiency. The model has already been deployed online, serving the search traffic of Taobao for over a year.