# Beyond Relevance: Structured Semantic Supervision for Product Search with LLM-Augmented Annotations

arXiv: https://arxiv.org/abs/2609.23646 · HTML: https://mlsanity.com/paper/2609.23646/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 42 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [PLAID: An Efficient Engine for Late Interaction Retrieval](https://mlsanity.com/paper/2205.09707/) (2022)
- [Inferring Networks of Substitutable and Complementary Products](https://mlsanity.com/paper/1506.08839/) (2015)
- [Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search](https://mlsanity.com/paper/2206.06588/) (2022)
- [Embedding-based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2106.09297/) (2021)
- [Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval](https://mlsanity.com/paper/1805.07591/) (2018)
- [On Application of Learning to Rank for E-Commerce Search](https://mlsanity.com/paper/1903.04263/) (2019)
- [SLIM: Sparsified Late Interaction for Multi-Vector Retrieval with Inverted Indexes](https://mlsanity.com/paper/2302.06587/) (2023)
- [Exploring the Viability of Synthetic Query Generation for Relevance Prediction](https://mlsanity.com/paper/2305.11944/) (2023)
- [Domain Adaptation of Foundation LLMs for e-Commerce](https://mlsanity.com/paper/2501.09706/) (2025)
- [Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search](https://mlsanity.com/paper/2502.15990/) (2025)
- [A Semantic Alignment System for Multilingual Query-Product Retrieval](https://mlsanity.com/paper/2208.02958/) (2022)
- [Centrality-aware Product Retrieval and Ranking](https://mlsanity.com/paper/2410.15930/) (2024)
- [Overview of the TREC 2023 Product Product Search Track](https://mlsanity.com/paper/2311.07861/) (2023)
- [Aug2Search: Enhancing Facebook Marketplace Search with LLM-Generated Synthetic Data Augmentation](https://mlsanity.com/paper/2505.16065/) (2025)
- [Intent term selection and refinement in e-commerce queries](https://mlsanity.com/paper/1908.08564/) (2019)
- [Synthetic Data Powers Product Retrieval for Long-tail Knowledge-Intensive Queries in E-commerce Search](https://mlsanity.com/paper/2602.23620/) (2026)

## Abstract (arXiv)

E-commerce search requires distinguishing products that are merely related to a query from those that directly satisfy the user's shopping intent. We augment query-product pairs with structured LLM-generated query and product attributes and human-validated relevance, explanations, and centrality judgments, and evaluate these signals using a simple dual-encoder retriever and MLP re-ranker. On an augmented subset of ESCI, a human-feature oracle reaches 0.9382 nDCG@10, while a human-free trained Q+P configuration reaches 0.9258. Synthetic approximations of the human signals reach 0.9150 overall but provide substantial gains for difficult, low-performing queries. Ablations show that most of the oracle improvement comes from post-edited explanations and annotator comments rather than the scalar centrality feature, suggesting that LLMs are most useful for exposing and approximating structured semantic supervision rather than replacing human judgment directly.