# Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models

arXiv: https://arxiv.org/abs/2105.04651 · HTML: https://mlsanity.com/paper/2105.04651/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2021.

## Built on by (most cited first, 18 of 18)

- [Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers](https://mlsanity.com/paper/2311.09180/) (2023)
- [Joint Multisided Exposure Fairness for Recommendation](https://mlsanity.com/paper/2205.00048/) (2022)
- [Joint Optimization of Ranking and Calibration with Contextualized Hybrid Model](https://mlsanity.com/paper/2208.06164/) (2022)
- [Leveraging Vision-Language Models for Granular Market Change Prediction](https://mlsanity.com/paper/2301.10166/) (2023)
- [Ranked List Truncation for Large Language Model-based Re-Ranking](https://mlsanity.com/paper/2404.18185/) (2024)
- [AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking](https://mlsanity.com/paper/2505.18512/) (2025)
- [Stability and Multigroup Fairness in Ranking with Uncertain Predictions](https://mlsanity.com/paper/2402.09326/) (2024)
- [Multivariate Representation Learning for Information Retrieval](https://mlsanity.com/paper/2304.14522/) (2023)
- [Uncertainty Calibration for Deep Audio Classifiers](https://mlsanity.com/paper/2206.13071/) (2022)
- [Efficient Uncertainty Estimation with Gaussian Process for Reliable Dialog Response Retrieval](https://mlsanity.com/paper/2303.08599/) (2023)
- [Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection](https://mlsanity.com/paper/2309.06131/) (2023)
- [Ranking Distillation for Open-Ended Video Question Answering with Insufficient Labels](https://mlsanity.com/paper/2403.14430/) (2024)
- [On the Calibration and Uncertainty with P'olya-Gamma Augmentation for Dialog Retrieval Models](https://mlsanity.com/paper/2303.08606/) (2023)
- [Bridging Personalization and Control in Scientific Personalized Search](https://mlsanity.com/paper/2411.02790/) (2024)
- [Are Recommenders Self-Aware? Label-Free Recommendation Performance Estimation via Model Uncertainty](https://mlsanity.com/paper/2507.23208/) (2025)
- [From Noise to Order: Learning to Rank via Denoising Diffusion](https://mlsanity.com/paper/2602.11453/) (2026)
- [Quantifying Divergence in Inter-LLM Communication Through API Retrieval and Ranking](https://mlsanity.com/paper/2604.22760/) (2026)
- [Can LLMs Take Retrieved Information with a Grain of Salt?](https://mlsanity.com/paper/2605.06919/) (2026)

## Abstract (arXiv)

In any ranking system, the retrieval model outputs a single score for a document based on its belief on how relevant it is to a given search query. While retrieval models have continued to improve with the introduction of increasingly complex architectures, few works have investigated a retrieval model's belief in the score beyond the scope of a single value. We argue that capturing the model's uncertainty with respect to its own scoring of a document is a critical aspect of retrieval that allows for greater use of current models across new document distributions, collections, or even improving effectiveness for down-stream tasks. In this paper, we address this problem via an efficient Bayesian framework for retrieval models which captures the model's belief in the relevance score through a stochastic process while adding only negligible computational overhead. We evaluate this belief via a ranking based calibration metric showing that our approximate Bayesian framework significantly improves a retrieval model's ranking effectiveness through a risk aware reranking as well as its confidence calibration. Lastly, we demonstrate that this additional uncertainty information is actionable and reliable on down-stream tasks represented via cutoff prediction.