# An Analysis of Variations in the Effectiveness of Query Performance Prediction

arXiv: https://arxiv.org/abs/2202.06306 · HTML: https://mlsanity.com/paper/2202.06306/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2022.

## Built on by (most cited first, 5 of 5)

- [Query Performance Prediction using Relevance Judgments Generated by Large Language Models](https://mlsanity.com/paper/2404.01012/) (2024)
- [Query Performance Prediction: From Ad-hoc to Conversational Search](https://mlsanity.com/paper/2305.10923/) (2023)
- [Predicting Retrieval Utility and Answer Quality in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.14546/) (2026)
- [Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines](https://mlsanity.com/paper/2604.22661/) (2026)
- [RAQG-QPP: Query Performance Prediction with Retrieved Query Variants and Retrieval Augmented Query Generation](https://mlsanity.com/paper/2604.27244/) (2026)

## Abstract (arXiv)

A query performance predictor estimates the retrieval effectiveness of an IR system for a given query. An important characteristic of QPP evaluation is that, since the ground truth retrieval effectiveness for QPP evaluation can be measured with different metrics, the ground truth itself is not absolute, which is in contrast to other retrieval tasks, such as that of ad-hoc retrieval. Motivated by this argument, the objective of this paper is to investigate how such variances in the ground truth for QPP evaluation can affect the outcomes of QPP experiments. We consider this not only in terms of the absolute values of the evaluation metrics being reported (e.g. Pearson's r, Kendall's τ), but also with respect to the changes in the ranks of different QPP systems when ordered by the QPP metric scores. Our experiments reveal that the observed QPP outcomes can vary considerably, both in terms of the absolute evaluation metric values and also in terms of the relative system ranks. Through our analysis, we report the optimal combinations of QPP evaluation metric and experimental settings that are likely to lead to smaller variations in the observed results.