mlsanityData to 14 Oct 2026

Large language models can accurately predict searcher preferences

arXiv 2309.10621 ↗First posted 19 Sep 2023cs.IR cs.AI cs.CLcited by 131

Not on the map. Why?

Abstract (arXiv)

Relevance labels, which indicate whether a search result is valuable to a searcher, are key to evaluating and optimising search systems. The best way to capture the true preferences of users is to ask them for their careful feedback on which results would be useful, but this approach does not scale to produce a large number of labels. Getting relevance labels at scale is usually done with third-party labellers, who judge on behalf of the user, but there is a risk of low-quality data if the labeller doesn't understand user needs. To improve quality, one standard approach is to study real users through interviews, user studies and direct feedback, find areas where labels are systematically disagreeing with users, then educate labellers about user needs through judging guidelines, training and monitoring. This paper introduces an alternate approach for improving label quality. It takes careful feedback from real users, which by definition is the highest-quality first-party gold data that can be derived, and develops an large language model prompt that agrees with that data. We present ideas and observations from deploying language models for large-scale relevance labelling at Bing, and illustrate with data from TREC. We have found large language models can be effective, with accuracy as good as human labellers and similar capability to pick the hardest queries, best runs, and best groups. Systematic changes to the prompts make a difference in accuracy, but so too do simple paraphrases. To measure agreement with real searchers needs high-quality "gold" labels, but with these we find that models produce better labels than third-party workers, for a fraction of the cost, and these labels let us train notably better rankers.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Builds on 23 of its 62 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

23 of 62 references.

References · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  6. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  8. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  10. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
11 more
  1. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  2. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  3. 2022The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrinkcited by 184
  4. 2022Measuring the Carbon Intensity of AI in Cloud Instancescited by 111
  5. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 100
  6. 2013Discrimination in Online Ad Deliverycited by 93
  7. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
  8. 2020The Problem with Metrics is a Fundamental Problem for AIcited by 37
  9. 2020Local Self-Attention over Long Text for Efficient Document Retrievalcited by 18
  10. 2023Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collectioncited by 5
  11. 2022Less is Less: When Are Snippets Insufficient for Human vs Machine Relevance Estimation?cited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 47; 2025: 41; 2026: 39.

131 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
  3. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  4. 2024Can LLMs Understand Time Series Anomalies?cited by 41
  5. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  6. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  7. 2024Rethinking Large Language Model Architectures for Sequential Recommendationscited by 27
  8. 2024TnT-LLM: Text Mining at Scale with Large Language Modelscited by 26
  9. 2025A Survey on Privacy Risks and Protection in Large Language Modelscited by 21
  10. 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
  11. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 20
  12. 2023Large Language Models for Propaganda Span Annotationcited by 19
38 more
  1. 2024FaithLM: Towards Faithful Explanations for Large Language Modelscited by 19
  2. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  3. 2024Synthetic Test Collections for Retrieval Evaluationcited by 19
  4. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  5. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  6. 2024Large Action Models: From Inception to Implementationcited by 18
  7. 2024Corpus-Steered Query Expansion with Large Language Modelscited by 17
  8. 2024Can We Use Large Language Models to Fill Relevance Judgment Holes?cited by 17
  9. 2024Knowledge Navigator: LLM-guided Browsing Framework for Exploratory Search in Scientific Literaturecited by 17
  10. 2024StraGo: Harnessing Strategic Guidance for Prompt Optimizationcited by 17
  11. 2025User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluationcited by 17
  12. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
  13. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  14. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  15. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  16. 2023Advancing the Search Frontier with AI Agentscited by 12
  17. 2024From Prompt Engineering to Prompt Science With Human in the Loopcited by 12
  18. 2024Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Gamescited by 12
  19. 2024Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluationcited by 12
  20. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  21. 2024Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Frameworkcited by 11
  22. 2024AMPO: Automatic Multi-Branched Prompt Optimizationcited by 11
  23. 2023It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Predictioncited by 10
  24. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agentscited by 10
  25. 2024Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.Icited by 9
  26. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAGcited by 9
  27. 2026Evaluating and Benchmarking the System One Model Jevcited by 9
  28. 2024Large Language Models for Relevance Judgment in Product Searchcited by 8
  29. 2024An Investigation of Prompt Variations for Zero-shot LLM-based Rankerscited by 8
  30. 2025Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggetscited by 8
  31. 2024ReSLLM: Large Language Models are Strong Resource Selectors for Federated Searchcited by 7
  32. 2024Generative Information Retrieval Evaluationcited by 7
  33. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7
  34. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  35. 2025mFollowIR: a Multilingual Benchmark for Instruction Following in Retrievalcited by 7
  36. 2025How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspectivecited by 7
  37. 2025Towards Understanding Bias in Synthetic Data for Evaluationcited by 7
  38. 2024Sociotechnical Implications of Generative Artificial Intelligence for Information Accesscited by 6

50 most cited of 131 All 131 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Search
  2. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
  3. 2026Aligning Large Language Models with Searcher Preferences
  4. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
  5. 2026RAC: Retrieval-Augmented Clarification for Faithful Conversational Search
  6. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking
  7. 2024RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine
  8. 2024Corpus-informed Retrieval Augmented Generation of Clarifying Questions