mlsanityData to 14 Oct 2026

To Judge or not to Judge: Using LLM Judgements for Advertiser Keyphrase Relevance at eBay

arXiv 2505.04209 ↗First posted 7 May 2025cs.IR cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

E-commerce sellers are recommended keyphrases based on their inventory on which they advertise to increase buyer engagement (clicks/sales). The relevance of advertiser keyphrases plays an important role in preventing the inundation of search systems with numerous irrelevant items that compete for attention in auctions, in addition to maintaining a healthy seller perception. In this work, we describe the shortcomings of training Advertiser keyphrase relevance filter models on click/sales/search relevance signals and the importance of aligning with human judgment, as sellers have the power to adopt or reject said keyphrase recommendations. In this study, we frame Advertiser keyphrase relevance as a complex interaction between 3 dynamical systems -- seller judgment, which influences seller adoption of our product, Advertising, which provides the keyphrases to bid on, and Search, who holds the auctions for the same keyphrases. This study discusses the practicalities of using human judgment via a case study at eBay Advertising and demonstrate that using LLM-as-a-judge en-masse as a scalable proxy for seller judgment to train our relevance models achieves a better harmony across the three systems -- provided that they are bound by a meticulous evaluation framework grounded in business metrics.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 7 May 2025.

Builds on

25 of 42 references.

References · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024A Survey on LLM-as-a-Judgecited by 1,423
  4. 2023Large Language Modelscited by 780
  5. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  6. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  7. 2020Bias and Debias in Recommender System: A Survey and Future Directionscited by 240
  8. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 157
  9. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  10. 2023Large language models can accurately predict searcher preferencescited by 131
  11. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 120
  12. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
13 more
  1. 2021Generalization in NLI: Ways (Not) To Go Beyond Simple Heuristicscited by 67
  2. 2024Large Language Models as Recommender Systems: A Study of Popularity Biascited by 26
  3. 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
  4. 2024Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Modelscited by 23
  5. 2022On Modality Bias Recognition and Reductioncited by 20
  6. 2023Rec4Ad: A Free Lunch to Mitigate Sample Selection Bias for Ads CTR Prediction in Taobaocited by 11
  7. 2024Improving Pinterest Search Relevance Using Large Language Modelscited by 8
  8. 2023An Offline Metric for the Debiasedness of Click Modelscited by 6
  9. 2024Graphite: A Graph-based Extreme Multi-Label Short Text Classifier for Keyphrase Recommendationcited by 6
  10. 2024From Lazy to Prolific: Tackling Missing Labels in Open Vocabulary Extreme Classification by Positive-Unlabeled Sequence Learningcited by 5
  11. 2024GraphEx: A Graph-based Extraction Method for Advertiser Keyphrase Recommendationcited by 3
  12. 2025Middleman Bias in Advertising: Aligning Relevance of Keyphrase Recommendations with Searchcited by 3
  13. 2025DashCLIP: Leveraging multimodal models for generating semantic embeddings for DoorDashcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBaycited by 2
  2. 2025BroadGen: A Framework for Generating Effective and Efficient Advertiser Broad Match Keyphrase Recommendationscited by 1
  3. 2026LLM-OSDA: An Optimal-Stopping Dynamic Auction for Native Advertising in Multi-Turn LLM Conversationscited by 0
  4. 2026A Systematic Investigation of Bias in Large Language Models for Advertising Relevancecited by 0