# Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing

arXiv: https://arxiv.org/abs/2404.11791 · HTML: https://mlsanity.com/paper/2404.11791/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2024.

## Built on by (most cited first, 9 of 9)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models](https://mlsanity.com/paper/2502.18817/) (2025)
- [Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information](https://mlsanity.com/paper/2506.10859/) (2025)
- [Generative Product Recommendations for Implicit Superlative Queries](https://mlsanity.com/paper/2504.18748/) (2025)
- [Optimizing Compound Retrieval Systems](https://mlsanity.com/paper/2504.12063/) (2025)
- [Post Hoc Regression Refinement via Pairwise Rankings](https://mlsanity.com/paper/2508.16495/) (2025)
- [Answer First, Reason Later: Aligning Search Relevance via Mode-Balanced Reinforcement Learning](https://mlsanity.com/paper/2602.10006/) (2026)
- [Scientific Discovery under Validation Congestion via Multi-Fidelity Pairwise Rankings](https://mlsanity.com/paper/2610.01827/) (2026)

## Abstract (arXiv)

The powerful generative abilities of large language models (LLMs) show potential in generating relevance labels for search applications. Previous work has found that directly asking about relevancy, such as ``How relevant is document A to query Q?", results in sub-optimal ranking. Instead, the pairwise ranking prompting (PRP) approach produces promising ranking performance through asking about pairwise comparisons, e.g., ``Is document A more relevant than document B to query Q?". Thus, while LLMs are effective at their ranking ability, this is not reflected in their relevance label generation. In this work, we propose a post-processing method to consolidate the relevance labels generated by an LLM with its powerful ranking abilities. Our method takes both LLM generated relevance labels and pairwise preferences. The labels are then altered to satisfy the pairwise preferences of the LLM, while staying as close to the original values as possible. Our experimental results indicate that our approach effectively balances label accuracy and ranking performance. Thereby, our work shows it is possible to combine both the ranking and labeling abilities of LLMs through post-processing.