Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processing
Not on the map. Why?
Abstract (arXiv)
Query Performance Prediction (QPP) estimates retrieval systems effectiveness for a given query, offering valuable insights for search effectiveness and query processing. Despite extensive research, QPPs face critical challenges in generalizing across diverse retrieval paradigms and collections. This paper provides a comprehensive evaluation of state-of-the-art QPPs (e.g. NQC, UQC), LETOR-based features, and newly explored dense-based predictors. Using diverse sparse rankers (BM25, DFree without and with query expansion) and hybrid or dense (SPLADE and ColBert) rankers and diverse test collections ROBUST, GOV2, WT10G, and MS MARCO; we investigate the relationships between predicted and actual performance, with a focus on generalization and robustness. Results show significant variability in predictors accuracy, with collections as the main factor and rankers next. Some sparse predictors perform somehow on some collections (TREC ROBUST and GOV2) but do not generalise to other collections (WT10G and MS-MARCO). While some predictors show promise in specific scenarios, their overall limitations constrain their utility for applications. We show that QPP-driven selective query processing offers only marginal gains, emphasizing the need for improved predictors that generalize across collections, align with dense retrieval architectures and are useful for downstream applications.
Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Builds on 24 of its 83 references that are in the map. First posted on arXiv 1 Apr 2025.
Builds on
24 of 83 references.
References · 24
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,388
- 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,237
- 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,169
- 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 468
- 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 284
- 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 127
- 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
- 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
- 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
- 2020Distilling Dense Representations for Ranking using Tightly-Coupled Teacherscited by 52
- 2017Neural Models for Information Retrievalcited by 29
- 2024Query Performance Prediction using Relevance Judgments Generated by Large Language Modelscited by 18
12 more
- 2023Query Performance Prediction: From Ad-hoc to Conversational Searchcited by 16
- 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
- 2023Query-specific Variable Depth Pooling via Query Performance Prediction towards Reducing Relevance Assessment Effortcited by 6
- 2025Judging the Judges: A Collection of LLM-Generated Relevance Judgementscited by 6
- 2019Forward and Backward Feature Selection for Query Performance Predictioncited by 5
- 2023Query Performance Prediction for Neural IR: Are We There Yet?cited by 5
- 2024A Deep Learning Approach for Selective Relevance Feedbackcited by 5
- 2022Groupwise Query Performance Prediction with BERTcited by 4
- 2023Selective Query Processing: a Risk-Sensitive Selection of System Configurationscited by 3
- 2023On Coherence-based Predictors for Dense Query Performance Predictioncited by 3
- 2024Can we predict QPP? An approach based on multivariate outlierscited by 2
- 2025Combining Query Performance Predictors: A Reproducibility Studycited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.