# Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search

arXiv: https://arxiv.org/abs/2206.06588 · HTML: https://mlsanity.com/paper/2206.06588/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 16 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [Inferring Networks of Substitutable and Complementary Products](https://mlsanity.com/paper/1506.08839/) (2015)
- [TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime](https://mlsanity.com/paper/2104.09399/) (2021)

## Built on by (most cited first, 30 of 65)

- [eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data](https://mlsanity.com/paper/2402.08831/) (2024)
- [KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model](https://mlsanity.com/paper/2506.20923/) (2025)
- [Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learning](https://mlsanity.com/paper/2503.24289/) (2025)
- [What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts](https://mlsanity.com/paper/2505.13360/) (2025)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs](https://mlsanity.com/paper/2509.20758/) (2025)
- [Language Models As Semantic Indexers](https://mlsanity.com/paper/2310.07815/) (2023)
- [Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models](https://mlsanity.com/paper/2410.20745/) (2024)
- [JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resources](https://mlsanity.com/paper/2407.20750/) (2024)
- [F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual World](https://mlsanity.com/paper/2603.19223/) (2026)
- [Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomy](https://mlsanity.com/paper/2403.04160/) (2024)
- [C-SEO Bench: Does Conversational SEO Work?](https://mlsanity.com/paper/2506.11097/) (2025)
- [It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction](https://mlsanity.com/paper/2311.07930/) (2023)
- [On the Impact of Feature Heterophily on Link Prediction with Graph Neural Networks](https://mlsanity.com/paper/2409.17475/) (2024)
- [Exploring the Viability of Synthetic Query Generation for Relevance Prediction](https://mlsanity.com/paper/2305.11944/) (2023)
- [Improving Pinterest Search Relevance Using Large Language Models](https://mlsanity.com/paper/2410.17152/) (2024)
- [Investigating LLM Applications in E-Commerce](https://mlsanity.com/paper/2408.12779/) (2024)
- [Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Data](https://mlsanity.com/paper/2410.17337/) (2024)
- [AgenticGEO: A Self-Evolving Agentic System for Generative Engine Optimization](https://mlsanity.com/paper/2603.20213/) (2026)
- [Generative Product Recommendations for Implicit Superlative Queries](https://mlsanity.com/paper/2504.18748/) (2025)
- [EcomMMMU: Strategic Utilization of Visuals for Robust Multimodal E-commerce Models](https://mlsanity.com/paper/2508.15721/) (2025)
- [Multi-Field Adaptive Retrieval](https://mlsanity.com/paper/2410.20056/) (2024)
- [Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe](https://mlsanity.com/paper/2509.16411/) (2025)
- [Purely Semantic Indexing for LLM-based Generative Recommendation and Retrieval](https://mlsanity.com/paper/2509.16446/) (2025)
- [Semantic Equivalence of e-Commerce Queries](https://mlsanity.com/paper/2308.03869/) (2023)
- [Situated Natural Language Explanations](https://mlsanity.com/paper/2308.14115/) (2023)
- [Overview of the TREC 2023 Product Product Search Track](https://mlsanity.com/paper/2311.07861/) (2023)
- [Generative Representational Learning of Foundation Models for Recommendation](https://mlsanity.com/paper/2506.11999/) (2025)
- [Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Search](https://mlsanity.com/paper/2507.05006/) (2025)
- [Thinking Broad, Acting Fast: Latent Reasoning Distillation from Multi-Perspective Chain-of-Thought for E-Commerce Relevance](https://mlsanity.com/paper/2601.21611/) (2026)

## Abstract (arXiv)

Improving the quality of search results can significantly enhance users experience and engagement with search engines. In spite of several recent advancements in the fields of machine learning and data mining, correctly classifying items for a particular user search query has been a long-standing challenge, which still has a large room for improvement. This paper introduces the "Shopping Queries Dataset", a large dataset of difficult Amazon search queries and results, publicly released with the aim of fostering research in improving the quality of search results. The dataset contains around 130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements. The dataset is multilingual with queries in English, Japanese, and Spanish. The Shopping Queries Dataset is being used in one of the KDDCup'22 challenges. In this paper, we describe the dataset and present three evaluation tasks along with baseline results: (i) ranking the results list, (ii) classifying product results into relevance categories, and (iii) identifying substitute products for a given query. We anticipate that this data will become the gold standard for future research in the topic of product search.