# Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels

arXiv: https://arxiv.org/abs/2310.14122 · HTML: https://mlsanity.com/paper/2310.14122/

## Key facts

- Cited by 86 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2023.

## Built on by (most cited first, 30 of 86)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://mlsanity.com/paper/2403.20327/) (2024)
- [Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning](https://mlsanity.com/paper/2503.06034/) (2025)
- [STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases](https://mlsanity.com/paper/2404.13207/) (2024)
- [TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy](https://mlsanity.com/paper/2406.11678/) (2024)
- [Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers](https://mlsanity.com/paper/2410.02642/) (2024)
- [CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking](https://mlsanity.com/paper/2412.01007/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [Semantic Operators: A Declarative Model for Rich, AI-based Data Processing](https://mlsanity.com/paper/2407.11418/) (2024)
- [Are Large Language Models Good at Utility Judgments?](https://mlsanity.com/paper/2403.19216/) (2024)
- [MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval](https://mlsanity.com/paper/2506.12364/) (2025)
- [Ranked List Truncation for Large Language Model-based Re-Ranking](https://mlsanity.com/paper/2404.18185/) (2024)
- [FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generation](https://mlsanity.com/paper/2402.11891/) (2024)
- [Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmark](https://mlsanity.com/paper/2406.01607/) (2024)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [Do Large Language Models Rank Fairly? An Empirical Study on the Fairness of LLMs as Rankers](https://mlsanity.com/paper/2404.03192/) (2024)
- [DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task](https://mlsanity.com/paper/2406.16332/) (2024)
- [BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives](https://mlsanity.com/paper/2402.14151/) (2024)
- [A Human-AI Comparative Analysis of Prompt Sensitivity in LLM-Based Relevance Judgment](https://mlsanity.com/paper/2504.12408/) (2025)
- [Benchmarking LLM-based Relevance Judgment Methods](https://mlsanity.com/paper/2504.12558/) (2025)
- [AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark](https://mlsanity.com/paper/2412.13102/) (2024)
- [LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking](https://mlsanity.com/paper/2504.07439/) (2025)
- [Evaluating Generative Ad Hoc Information Retrieval](https://mlsanity.com/paper/2311.04694/) (2023)
- [ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting](https://mlsanity.com/paper/2501.06582/) (2025)
- [Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search](https://mlsanity.com/paper/2512.00968/) (2025)
- [It's All Relative! -- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction](https://mlsanity.com/paper/2311.07930/) (2023)
- [CoRanking: Collaborative Ranking with Small and Large Ranking Agents](https://mlsanity.com/paper/2503.23427/) (2025)
- [Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I](https://mlsanity.com/paper/2407.02464/) (2024)
- [ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation](https://mlsanity.com/paper/2410.05168/) (2024)
- [Matchmaker: Self-Improving Large Language Model Programs for Schema Matching](https://mlsanity.com/paper/2410.24105/) (2024)

## Abstract (arXiv)

Zero-shot text rankers powered by recent LLMs achieve remarkable ranking performance by simply prompting. Existing prompts for pointwise LLM rankers mostly ask the model to choose from binary relevance labels like "Yes" and "No". However, the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers for documents that are partially relevant to the query. We propose to incorporate fine-grained relevance labels into the prompt for LLM rankers, enabling them to better differentiate among documents with different levels of relevance to the query and thus derive a more accurate ranking. We study two variants of the prompt template, coupled with different numbers of relevance levels. Our experiments on 8 BEIR data sets show that adding fine-grained relevance labels significantly improves the performance of LLM rankers.