mlsanityData to 7 Oct 2026

Perspectives on Large Language Models for Relevance Judgment

arXiv 2304.09161 ↗First posted 13 Apr 2023cs.IR cs.CYcited by 84

Not on the map. Why?

Abstract (arXiv)

When asked, large language models (LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in evaluations of retrieval systems. In this perspectives paper, we discuss possible ways for LLMs to support relevance judgments along with concerns and issues that arise. We devise a human--machine collaboration spectrum that allows to categorize different relevance judgment strategies, based on how much humans rely on machines. For the extreme point of "fully automated judgments", we further include a pilot experiment on whether LLM-based relevance judgments correlate with judgments from trained human assessors. We conclude the paper by providing opposing perspectives for and against the use of~LLMs for automatic relevance judgments, and a compromise perspective, informed by our analyses of the literature, our preliminary experimental evidence, and our experience as IR researchers.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Apr 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 24; 2025: 27; 2026: 23.

84 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  3. 2023Large language models can accurately predict searcher preferencescited by 129
  4. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  5. 2023AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Riskscited by 34
  6. 2023Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomiescited by 34
  7. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  8. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 31
  9. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  10. 2024Personas with Attitudes: Controlling LLMs for Diverse Data Annotationcited by 21
  11. 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
  12. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
38 more
  1. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  2. 2024Synthetic Test Collections for Retrieval Evaluationcited by 18
  3. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  4. 2024Knowledge Navigator: LLM-guided Browsing Framework for Exploratory Search in Scientific Literaturecited by 17
  5. 2024Leveraging Large Language Models for Relevance Judgments in Legal Case Retrievalcited by 16
  6. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  7. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  8. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  9. 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
  10. 2023Advancing the Search Frontier with AI Agentscited by 12
  11. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  12. 2024Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Frameworkcited by 11
  13. 2024Consistent estimation of generative model representations in the data kernel perspective spacecited by 10
  14. 2023It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Predictioncited by 9
  15. 2024Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.Icited by 9
  16. 2025Do we still need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Predictioncited by 9
  17. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agentscited by 9
  18. 2024Large Language Models for Relevance Judgment in Product Searchcited by 8
  19. 2025Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggetscited by 8
  20. 2024ReSLLM: Large Language Models are Strong Resource Selectors for Federated Searchcited by 7
  21. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7
  22. 2025Towards Understanding Bias in Synthetic Data for Evaluationcited by 7
  23. 2025How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspectivecited by 6
  24. 2025Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Informationcited by 6
  25. 2023DepreSym: A Depression Symptom Annotated Corpus and the Role of LLMs as Assessors of Psychological Markerscited by 5
  26. 2024ScholarChemQA: Unveiling the Power of Language Models in Chemical Research Question Answeringcited by 5
  27. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluationcited by 5
  28. 2025Automated Query-Product Relevance Labeling using Large Language Models for E-commerce Searchcited by 5
  29. 2025Generative Product Recommendations for Implicit Superlative Queriescited by 5
  30. 2025Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevancecited by 4
  31. 2023Revisiting Automated Topic Model Evaluation with Large Language Modelscited by 3
  32. 2024Best in Tau@LLMJudge: Criteria-Based Relevance Evaluation with Llama3cited by 3
  33. 2024Boosting LLM-based Relevance Modeling with Distribution-Aware Robust Learningcited by 3
  34. 2026Evaluating and Benchmarking the System One Model Jevcited by 3
  35. 2024The Decoy Dilemma in Online Medical Information Evaluation: A Comparative Study of Credibility Assessments by LLM and Human Judgescited by 2
  36. 2025Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processingcited by 2
  37. 2025Augmented Relevance Datasets with Fine-Tuned Small LLMscited by 2
  38. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2

50 most cited of 84 All 84 (JSON)