# LLM-based User-Product Relevance Evaluation for Dynamic Product Advertising

arXiv: https://arxiv.org/abs/2610.14550 · HTML: https://mlsanity.com/paper/2610.14550/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Builds on (most cited first, 7 of 7)

- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment](https://mlsanity.com/paper/2502.18965/) (2025)
- [Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search](https://mlsanity.com/paper/2206.06588/) (2022)
- [Judging the Judges: A Collection of LLM-Generated Relevance Judgements](https://mlsanity.com/paper/2502.13908/) (2025)
- [Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce](https://mlsanity.com/paper/1809.03186/) (2018)
- [BloomIntent: Automating Search Evaluation with LLM-Generated Fine-Grained User Intents](https://mlsanity.com/paper/2509.18641/) (2025)
- [Semantic Ads Retrieval at Walmart eCommerce with Language Models Progressively Trained on Multiple Knowledge Domains](https://mlsanity.com/paper/2502.09089/) (2025)

## Abstract (arXiv)

Dynamic Product Advertising (DPA) retrieval systems are a core component of modern e-commerce advertising; however, their evaluation remains challenging. Conventional offline engagement metrics are limited by sparse, biased feedback, while online evaluation is costly and time-consuming. In this work, we propose a user--product relevance metric for offline evaluation of DPA retrieval systems. The metric first compresses the engagement history into a compact set of intent representatives via Semantic-ID-grouped, event-stratified segmentation, then assesses the relevance of a recommended product to each individual intent and aggregates them as a user-level relevance score, yielding a stable and interpretable measure of retrieval quality. To compute this metric at scale, we leverage large language models (LLMs) as semantic relevance judges, capturing nuanced functional and semantic relationships that heuristic methods often miss. We propose an agentic, human-aligned prompting framework that generates multi-category relevance guidelines and distills teacher-model judgments into an efficient student model for scalable evaluation. The resulting metric strongly correlates with offline and online business metrics, enabling faster and more reliable DPA retrieval iteration before A/B testing.