mlsanityData to 14 Oct 2026

Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search

arXiv 2206.06588 ↗First posted 14 Jun 2022cs.IR cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

Improving the quality of search results can significantly enhance users experience and engagement with search engines. In spite of several recent advancements in the fields of machine learning and data mining, correctly classifying items for a particular user search query has been a long-standing challenge, which still has a large room for improvement. This paper introduces the "Shopping Queries Dataset", a large dataset of difficult Amazon search queries and results, publicly released with the aim of fostering research in improving the quality of search results. The dataset contains around 130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements. The dataset is multilingual with queries in English, Japanese, and Spanish. The Shopping Queries Dataset is being used in one of the KDDCup'22 challenges. In this paper, we describe the dataset and present three evaluation tasks along with baseline results: (i) ranking the results list, (ii) classifying product results into relevance categories, and (iii) identifying substitute products for a given query. We anticipate that this data will become the gold standard for future research in the topic of product search.

Key facts. Cited by 65 papers in the map (data through 14 Oct 2026). Builds on 8 of its 16 references that are in the map. First posted on arXiv 14 Jun 2022.

Builds on

8 of 16 references.

References · 8

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  5. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  6. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 204
  7. 2015Inferring Networks of Substitutable and Complementary Productscited by 92
  8. 2021TREC Deep Learning Track: Reusable Test Collections in the Large Data Regimecited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 12; 2025: 15; 2026: 29.

65 papers.

Sort:

By area:

  1. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 46
  2. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 36
  3. 2025Rec-R1: Bridging Generative Large Language Models and User-Centric Recommendation Systems via Reinforcement Learningcited by 33
  4. 2025What Prompts Don't Say: Understanding and Managing Underspecification in LLM Promptscited by 32
  5. 2025A Survey of Personalization: From RAG to Agentcited by 27
  6. 2025SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMscited by 24
  7. 2023Language Models As Semantic Indexerscited by 20
  8. 2024Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Modelscited by 20
  9. 2024JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resourcescited by 15
  10. 2026F2LLM-v2: Inclusive, Performant, and Efficient Embeddings for a Multilingual Worldcited by 12
  11. 2024Improving Retrieval in Theme-specific Applications using a Corpus Topical Taxonomycited by 11
  12. 2025C-SEO Bench: Does Conversational SEO Work?cited by 11
38 more
  1. 2023It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Predictioncited by 10
  2. 2024On the Impact of Feature Heterophily on Link Prediction with Graph Neural Networkscited by 10
  3. 2023Exploring the Viability of Synthetic Query Generation for Relevance Predictioncited by 9
  4. 2024Improving Pinterest Search Relevance Using Large Language Modelscited by 8
  5. 2024Investigating LLM Applications in E-Commercecited by 7
  6. 2024Captions Speak Louder than Images: Generalizing Foundation Models for E-commerce from High-quality Multimodal Instruction Datacited by 6
  7. 2026AgenticGEO: A Self-Evolving Agentic System for Generative Engine Optimizationcited by 6
  8. 2025Generative Product Recommendations for Implicit Superlative Queriescited by 5
  9. 2025EcomMMMU: Strategic Utilization of Visuals for Robust Multimodal E-commerce Modelscited by 5
  10. 2024Multi-Field Adaptive Retrievalcited by 4
  11. 2025Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipecited by 4
  12. 2025Purely Semantic Indexing for LLM-based Generative Recommendation and Retrievalcited by 3
  13. 2023Semantic Equivalence of e-Commerce Queriescited by 2
  14. 2023Situated Natural Language Explanationscited by 2
  15. 2023Overview of the TREC 2023 Product Product Search Trackcited by 2
  16. 2025Generative Representational Learning of Foundation Models for Recommendationcited by 2
  17. 2025Do We Really Need Specialization? Evaluating Generalist Text Embeddings for Zero-Shot Recommendation and Searchcited by 2
  18. 2026Thinking Broad, Acting Fast: Latent Reasoning Distillation from Multi-Perspective Chain-of-Thought for E-Commerce Relevancecited by 2
  19. 2026How Generative AI Disrupts Search: An Empirical Study of Google Search, Gemini, and AI Overviewscited by 2
  20. 2026GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimizationcited by 2
  21. 2026Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projectioncited by 2
  22. 2022Some Practice for Improving the Search Results of E-commercecited by 1
  23. 2024EC-Guide: A Comprehensive E-Commerce Guide for Instruction Tuning and Quantizationcited by 1
  24. 2026Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoderscited by 1
  25. 2026Policy Regret for Embedding Model Routing: Contextual Bandits with Low-Rank Expertscited by 1
  26. 2022Predicting Query-Item Relationship using Adversarial Training and Robust Modeling Techniquescited by 0
  27. 2023ZhichunRoad at Amazon KDD Cup 2022: MultiTask Pre-Training for E-Commerce Product Searchcited by 0
  28. 2024Large Language Model Driven Recommendationcited by 0
  29. 2024Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrievalcited by 0
  30. 2025Revisiting Graph Projections for Effective Complementary Product Recommendationcited by 0
  31. 2025RewardRank: Optimizing True Learning-to-Rank Utilitycited by 0
  32. 2025SEAL: Structure and Element Aware Learning to Improve Long Structured Document Retrievalcited by 0
  33. 2026EmbeddingRWKV: State-Centric Retrieval with Reusable Statescited by 0
  34. 2026Generalized Radius and Integrated Codebook Transforms for Differentiable Vector Quantizationcited by 0
  35. 2026RexBERT: Context Specialized Bidirectional Encoders for E-commercecited by 0
  36. 2026FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learningcited by 0
  37. 2026Semantic Retrieval for Product Search in E-Commercecited by 0
  38. 2026Customer-Agent: Overcoming Context Limitations in Ultra-Long Shopping Trajectories via Tool-Augmented Agents and RLVRcited by 0

50 most cited of 65 All 65 (JSON)