# Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing

arXiv: https://arxiv.org/abs/2504.01101 · HTML: https://mlsanity.com/paper/2504.01101/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 83 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking](https://mlsanity.com/paper/2107.05720/) (2021)
- [SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval](https://mlsanity.com/paper/2109.10086/) (2021)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [Distilling Dense Representations for Ranking using Tightly-Coupled Teachers](https://mlsanity.com/paper/2010.11386/) (2020)
- [Neural Models for Information Retrieval](https://mlsanity.com/paper/1705.01509/) (2017)
- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [Query Performance Prediction: From Ad-hoc to Conversational Search](https://mlsanity.com/paper/2305.10923/) (2023)
- [Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation](https://mlsanity.com/paper/2503.19092/) (2025)
- [Judging the Judges: A Collection of LLM-Generated Relevance Judgements](https://mlsanity.com/paper/2502.13908/) (2025)
- [Query-specific Variable Depth Pooling via Query Performance Prediction towards Reducing Relevance Assessment Effort](https://mlsanity.com/paper/2304.11752/) (2023)
- [Forward and Backward Feature Selection for Query Performance Prediction](https://mlsanity.com/paper/1912.04107/) (2019)
- [Query Performance Prediction for Neural IR: Are We There Yet?](https://mlsanity.com/paper/2302.09947/) (2023)
- [A Deep Learning Approach for Selective Relevance Feedback](https://mlsanity.com/paper/2401.11198/) (2024)
- [Groupwise Query Performance Prediction with BERT](https://mlsanity.com/paper/2204.11489/) (2022)
- [Selective Query Processing: a Risk-Sensitive Selection of System Configurations](https://mlsanity.com/paper/2305.18311/) (2023)
- [On Coherence-based Predictors for Dense Query Performance Prediction](https://mlsanity.com/paper/2310.11405/) (2023)
- [Can we predict QPP? An approach based on multivariate outliers](https://mlsanity.com/paper/2402.16875/) (2024)
- [Combining Query Performance Predictors: A Reproducibility Study](https://mlsanity.com/paper/2503.24251/) (2025)

## Built on by (most cited first, 2 of 2)

- [Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines](https://mlsanity.com/paper/2604.22661/) (2026)
- [Your Retriever Already Knows: Distribution-Shape QPP for RAG Retrieval Sufficiency](https://mlsanity.com/paper/2609.11646/) (2026)

## Abstract (arXiv)

Query Performance Prediction (QPP) estimates retrieval systems effectiveness for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, QPPs face critical challenges in generalizing across diverse retrieval paradigms and collections. This paper provides a comprehensive evaluation of state-of-the-art QPPs (e.g. NQC, UQC), LETOR-based features, and newly explored dense-based predictors. Using diverse sparse rankers (BM25, DFree without and with query expansion) and hybrid or dense (SPLADE and ColBert) rankers and diverse test collections ROBUST, GOV2, WT10G, and MS MARCO; we investigate the relationships between predicted and actual performance, with a focus on generalization and robustness. Results show significant variability in predictors accuracy, with collections as the main factor and rankers next. Some sparse predictors perform somehow on some collections (TREC ROBUST and GOV2) but do not generalise to other collections (WT10G and MS-MARCO). While some predictors show promise in specific scenarios, their overall limitations constrain their utility for applications. We show that QPP-driven selective query processing offers only marginal gains, emphasizing the need for improved predictors that generalize across collections, align with dense retrieval architectures and are useful for downstream applications.