# Leveraging LLMs for Unsupervised Dense Retriever Ranking

arXiv: https://arxiv.org/abs/2402.04853 · HTML: https://mlsanity.com/paper/2402.04853/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2024.

## Built on by (most cited first, 19 of 19)

- [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://mlsanity.com/paper/2403.20327/) (2024)
- [LLMs are Also Effective Embedding Models: An In-depth Overview](https://mlsanity.com/paper/2412.12591/) (2024)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [Can We Use Large Language Models to Fill Relevance Judgment Holes?](https://mlsanity.com/paper/2405.05600/) (2024)
- [DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.07233/) (2025)
- [AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark](https://mlsanity.com/paper/2412.13102/) (2024)
- [Retrieval-Enhanced Machine Learning: Synthesis and Opportunities](https://mlsanity.com/paper/2407.12982/) (2024)
- [Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information](https://mlsanity.com/paper/2506.10859/) (2025)
- [Query Understanding in LLM-based Conversational Information Seeking](https://mlsanity.com/paper/2504.06356/) (2025)
- [Do Audio-Language Models Understand Linguistic Variations?](https://mlsanity.com/paper/2410.16505/) (2024)
- [Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance](https://mlsanity.com/paper/2503.23239/) (2025)
- [LTRR: Learning To Rank Retrievers for LLMs](https://mlsanity.com/paper/2506.13743/) (2025)
- [Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales](https://mlsanity.com/paper/2505.19334/) (2025)
- [When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR](https://mlsanity.com/paper/2506.01877/) (2025)
- [Negative Matters: Multi-Granularity Hard-Negative Synthesis and Anchor-Token-Aware Pooling for Enhanced Text Embeddings](https://mlsanity.com/paper/2509.00842/) (2025)
- [LLM-VPRF: Large Language Model Based Vector Pseudo Relevance Feedback](https://mlsanity.com/paper/2504.01448/) (2025)
- [PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark](https://mlsanity.com/paper/2601.08363/) (2026)
- [Entriever: Energy-based Retriever for Knowledge-Grounded Dialog Systems](https://mlsanity.com/paper/2506.00585/) (2025)
- [Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding](https://mlsanity.com/paper/2608.07353/) (2026)

## Abstract (arXiv)

In this paper we present Large Language Model Assisted Retrieval Model Ranking (LARMOR), an effective unsupervised approach that leverages LLMs for selecting which dense retriever to use on a test corpus (target). Dense retriever selection is crucial for many IR applications that rely on using dense retrievers trained on public corpora to encode or search a new, private target corpus. This is because when confronted with domain shift, where the downstream corpora, domains, or tasks of the target corpus differ from the domain/task the dense retriever was trained on, its performance often drops. Furthermore, when the target corpus is unlabeled, e.g., in a zero-shot scenario, the direct evaluation of the model on the target corpus becomes unfeasible. Unsupervised selection of the most effective pre-trained dense retriever becomes then a crucial challenge. Current methods for dense retriever selection are insufficient in handling scenarios with domain shift. Our proposed solution leverages LLMs to generate pseudo-relevant queries, labels and reference lists based on a set of documents sampled from the target corpus. Dense retrievers are then ranked based on their effectiveness on these generated pseudo-relevant signals. Notably, our method is the first approach that relies solely on the target corpus, eliminating the need for both training corpora and test labels. To evaluate the effectiveness of our method, we construct a large pool of state-of-the-art dense retrievers. The proposed approach outperforms existing baselines with respect to both dense retriever selection and ranking. We make our code and results publicly available at https://github.com/ielab/larmor/.