# On the Calibration and Uncertainty with P'olya-Gamma Augmentation for Dialog Retrieval Models

arXiv: https://arxiv.org/abs/2303.08606 · HTML: https://mlsanity.com/paper/2303.08606/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 15 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks](https://mlsanity.com/paper/2002.10118/) (2020)
- [Masksembles for Uncertainty Estimation](https://mlsanity.com/paper/2012.08334/) (2020)
- [Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots](https://mlsanity.com/paper/2004.03588/) (2020)
- [On Sparse variational methods and the Kullback-Leibler divergence between stochastic processes](https://mlsanity.com/paper/1504.07027/) (2015)
- [Analyzing and Characterizing User Intent in Information-seeking Conversations](https://mlsanity.com/paper/1804.08759/) (2018)
- [Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selection](https://mlsanity.com/paper/2009.04703/) (2020)
- [Bayesian Few-Shot Classification with One-vs-Each P'olya-Gamma Augmented Gaussian Processes](https://mlsanity.com/paper/2007.10417/) (2020)
- [Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentation](https://mlsanity.com/paper/1506.05843/) (2015)
- [Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset](https://mlsanity.com/paper/1912.04639/) (2019)
- [GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning](https://mlsanity.com/paper/2102.07868/) (2021)
- [On the Calibration and Uncertainty of Neural Learning to Rank Models](https://mlsanity.com/paper/2101.04356/) (2021)
- [Transformer Uncertainty Estimation with Hierarchical Stochastic Attention](https://mlsanity.com/paper/2112.13776/) (2021)
- [Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Models](https://mlsanity.com/paper/2105.04651/) (2021)
- [Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentation](https://mlsanity.com/paper/1802.06383/) (2018)
- [Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation](https://mlsanity.com/paper/1905.09670/) (2019)
- ["None of the Above":Measure Uncertainty in Dialog Response Retrieval](https://mlsanity.com/paper/2004.01926/) (2020)
- [Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles](https://mlsanity.com/paper/2010.02586/) (2020)
- [Scalable Cross Validation Losses for Gaussian Process Models](https://mlsanity.com/paper/2105.11535/) (2021)
- [Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation](https://mlsanity.com/paper/1711.11486/) (2017)

## Abstract (arXiv)

Deep neural retrieval models have amply demonstrated their power but estimating the reliability of their predictions remains challenging. Most dialog response retrieval models output a single score for a response on how relevant it is to a given question. However, the bad calibration of deep neural network results in various uncertainty for the single score such that the unreliable predictions always misinform user decisions. To investigate these issues, we present an efficient calibration and uncertainty estimation framework PG-DRR for dialog response retrieval models which adds a Gaussian Process layer to a deterministic deep neural network and recovers conjugacy for tractable posterior inference by P'olya-Gamma augmentation. Finally, PG-DRR achieves the lowest empirical calibration error (ECE) in the in-domain datasets and the distributional shift task while keeping R10@1 and MAP performance.