mlsanityData to 14 Oct 2026

Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels

arXiv 2310.14122 ↗First posted 21 Oct 2023cs.IRcited by 86

Not on the map. Why?

Abstract (arXiv)

Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like "Yes" and "No". However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Oct 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 35; 2025: 27; 2026: 21.

86 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 83
  3. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 60
  4. 2024STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Basescited by 55
  5. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  6. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 43
  7. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  8. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  9. 2024Semantic Operators: A Declarative Model for Rich, AI-based Data Processingcited by 34
  10. 2024Are Large Language Models Good at Utility Judgments?cited by 22
  11. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  12. 2024Ranked List Truncation for Large Language Model-based Re-Rankingcited by 20
38 more
  1. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  2. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  3. 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
  4. 2024Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankerscited by 17
  5. 2024DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Taskcited by 17
  6. 2024BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectivescited by 15
  7. 2025A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgmentcited by 15
  8. 2025Benchmarking LLM-based Relevance Judgment Methodscited by 14
  9. 2024AIR-Bench: Automated Heterogeneous Information Retrieval Benchmarkcited by 12
  10. 2025LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Rerankingcited by 12
  11. 2023Evaluating Generative Ad Hoc Information Retrievalcited by 11
  12. 2025ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Draftingcited by 11
  13. 2025Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Searchcited by 11
  14. 2023It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Predictioncited by 10
  15. 2025CoRanking: Collaborative Ranking with Small and Large Ranking Agentscited by 10
  16. 2024Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.Icited by 9
  17. 2024ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillationcited by 9
  18. 2024Matchmaker: Self-Improving Large Language Model Programs for Schema Matchingcited by 9
  19. 2025Are LLMs Reliable Rankers? Rank Manipulation via Two-Stage Token Optimizationcited by 9
  20. 2023Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languagescited by 8
  21. 2024An Investigation of Prompt Variations for Zero-shot LLM-based Rankerscited by 8
  22. 2024STAR: A Simple Training-free Approach for Recommendations using Large Language Modelscited by 8
  23. 2025AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Rerankingcited by 8
  24. 2024ReSLLM: Large Language Models are Strong Resource Selectors for Federated Searchcited by 7
  25. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7
  26. 2024Self-Calibrated Listwise Reranking with Large Language Modelscited by 7
  27. 2025How do Large Language Models Understand Relevance? A Mechanistic Interpretability Perspectivecited by 7
  28. 2024W-RAG: Weakly Supervised Dense Retrieval in RAG for Open-domain Question Answeringcited by 6
  29. 2024RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Enginecited by 6
  30. 2025Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Informationcited by 6
  31. 2026Bayesian Active Learning with Gaussian Processes Guided by LLM Relevance Scoring for Dense Passage Retrievalcited by 6
  32. 2024Toward Automatic Relevance Judgment using Vision--Language Models for Image--Text Retrieval Evaluationcited by 5
  33. 2025Generative Product Recommendations for Implicit Superlative Queriescited by 5
  34. 2025SweRank+: Multilingual, Multi-Turn Code Ranking for Software Issue Localizationcited by 5
  35. 2024Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Modelscited by 4
  36. 2024Harnessing the Power of Semi-Structured Knowledge and LLMs with Triplet-Based Prefiltering for Question Answeringcited by 4
  37. 2024An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Rerankingcited by 4
  38. 2025How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Modelscited by 4

50 most cited of 86 All 86 (JSON)