# Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles

arXiv: https://arxiv.org/abs/2010.02586 · HTML: https://mlsanity.com/paper/2010.02586/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance](https://mlsanity.com/paper/2109.04349/) (2021)
- [Dialogue Term Extraction using Transfer Learning and Topological Data Analysis](https://mlsanity.com/paper/2208.10448/) (2022)
- [Dialogue Evaluation with Offline Reinforcement Learning](https://mlsanity.com/paper/2209.00876/) (2022)
- [A Confidence-based Acquisition Model for Self-supervised Active Learning and Label Correction](https://mlsanity.com/paper/2310.08944/) (2023)
- [Evaluating Predictive Uncertainty under Distributional Shift on Dialogue Dataset](https://mlsanity.com/paper/2109.00186/) (2021)
- [On the Calibration and Uncertainty with P'olya-Gamma Augmentation for Dialog Retrieval Models](https://mlsanity.com/paper/2303.08606/) (2023)
- [Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation](https://mlsanity.com/paper/2605.21949/) (2026)
- [When Retrieval Metrics Mislead: Measuring Policy Signal in Long-Horizon Tool-Use Agents](https://mlsanity.com/paper/2606.23937/) (2026)

## Abstract (arXiv)

The ability to accurately track what happens during a conversation is essential for the performance of a dialogue system. Current state-of-the-art multi-domain dialogue state trackers achieve just over 55% accuracy on the current go-to benchmark, which means that in almost every second dialogue turn they place full confidence in an incorrect dialogue state. Belief trackers, on the other hand, maintain a distribution over possible dialogue states. However, they lack in performance compared to dialogue state trackers, and do not produce well calibrated distributions. In this work we present state-of-the-art performance in calibration for multi-domain dialogue belief trackers using a calibrated ensemble of models. Our resulting dialogue belief tracker also outperforms previous dialogue belief tracking models in terms of accuracy.