mlsanityData to 7 Oct 2026

On the Calibration and Uncertainty with P'olya-Gamma Augmentation for Dialog Retrieval Models

arXiv 2303.08606 ↗First posted 15 Mar 2023cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep neural retrieval models have amply demonstrated their power but estimating the reliability of their predictions remains challenging. Most dialog response retrieval models output a single score for a response on how relevant it is to a given question. However, the bad calibration of deep neural network results in various uncertainty for the single score such that the unreliable predictions always misinform user decisions. To investigate these issues, we present an efficient calibration and uncertainty estimation framework PG-DRR for dialog response retrieval models which adds a Gaussian Process layer to a deterministic deep neural network and recovers conjugacy for tractable posterior inference by P'olya-Gamma augmentation. Finally, PG-DRR achieves the lowest empirical calibration error (ECE) in the in-domain datasets and the distributional shift task while keeping R10@1 and MAP performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 15 Mar 2023.

Builds on

24 of 35 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  5. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  6. 2020Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networkscited by 227
  7. 2020Masksembles for Uncertainty Estimationcited by 88
  8. 2020Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbotscited by 73
  9. 2015On Sparse variational methods and the Kullback-Leibler divergence between stochastic processescited by 56
  10. 2018Analyzing and Characterizing User Intent in Information-seeking Conversationscited by 36
  11. 2020Do Response Selection Models Really Know What's Next? Utterance Manipulation Strategies for Multi-turn Response Selectioncited by 36
  12. 2020Bayesian Few-Shot Classification with One-vs-Each P'olya-Gamma Augmented Gaussian Processescited by 32
12 more
  1. 2019Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Datasetcited by 26
  2. 2021GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learningcited by 26
  3. 2015Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentationcited by 25
  4. 2021On the Calibration and Uncertainty of Neural Learning to Rank Modelscited by 24
  5. 2021Transformer Uncertainty Estimation with Hierarchical Stochastic Attentioncited by 21
  6. 2021Not All Relevance Scores are Equal: Efficient Uncertainty and Calibration Modeling for Deep Retrieval Modelscited by 18
  7. 2018Efficient Gaussian Process Classification Using Polya-Gamma Data Augmentationcited by 17
  8. 2019Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentationcited by 16
  9. 2020Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensemblescited by 8
  10. 2020"None of the Above":Measure Uncertainty in Dialog Response Retrievalcited by 7
  11. 2021Scalable Cross Validation Losses for Gaussian Process Modelscited by 6
  12. 2017Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisationcited by 2

Built on by

Not cited yet.