mlsanityData to 7 Oct 2026

Can LLM Rerankers Predict Their Own Ranking Performance?

arXiv 2606.03535 ↗First posted 2 Jun 2026cs.IR cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Retrieval effectiveness varies substantially across queries, making it important to estimate ranking quality before relevance judgments are available. Query performance prediction (QPP) addresses this need, but most existing methods rely on external predictors after retrieval or reranking. In this paper, we study textitreranker-internal QPP: can an LLM reranker estimate the quality of the ranking it has just produced? We investigate both training-free and training-based approaches. For training-free estimation, we examine metric-specific self-consistency across sampled rankings and verbalized confidence produced directly by the reranker. Experiments on TREC Deep Learning 2019--2022 with four LLMs show that self-consistency is competitive with the state-of-the-art (SOTA) approach and better calibrated in almost all settings, while direct verbalized confidence is severely overconfident. To improve verbalized confidence, we propose two supervised methods, Verb-Num and Verb-List, which enable LLM rerankers to produce calibrated ranking-quality estimates with only a few additional output tokens.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 52 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

20 of 52 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  10. 2020Calibration of Pre-trained Transformerscited by 307
  11. 2022Prompting GPT-3 To Be Reliablecited by 258
  12. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
8 more
  1. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  2. 2024When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentationcited by 59
  3. 2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perceptioncited by 26
  4. 2025Overview of the TREC 2022 deep learning trackcited by 22
  5. 2022Deep-QPP: A Pairwise Interaction-based Deep Learning Model for Supervised Query Performance Predictioncited by 13
  6. 2024Sliding Windows Are Not the End: Exploring Full Ranking with Long-Context Large Language Modelscited by 8
  7. 2023On Coherence-based Predictors for Dense Query Performance Predictioncited by 3
  8. 2026Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answerscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems
  2. 2026RAQG-QPP: Query Performance Prediction with Retrieved Query Variants and Retrieval Augmented Query Generation
  3. 2026PRECISE: Reducing the Bias of LLM Evaluations Using Prediction-Powered Ranking Estimation
  4. 2026CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring
  5. 2025Rethinking LLM Parametric Knowledge as Post-retrieval Confidence for Dynamic Retrieval and Reranking
  6. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning
  7. 2026Think Thrice Before Reranking: Multi-perspective Evidence and Reasoning Integration for Text Reranking
  8. 2025LLM Optimization Unlocks Real-Time Pairwise Reranking