# Calibrating Structured Output Predictors for Natural Language Processing

arXiv: https://arxiv.org/abs/2004.04361 · HTML: https://mlsanity.com/paper/2004.04361/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Apr 2020.

## Built on by (most cited first, 17 of 17)

- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [Selectively Answering Ambiguous Questions](https://mlsanity.com/paper/2305.14613/) (2023)
- [Knowing More About Questions Can Help: Improving Calibration in Question Answering](https://mlsanity.com/paper/2106.01494/) (2021)
- [Uncertainty Estimation for Language Reward Models](https://mlsanity.com/paper/2203.07472/) (2022)
- [Calibrated Interpretation: Confidence Estimation in Semantic Parsing](https://mlsanity.com/paper/2211.07443/) (2022)
- [On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency](https://mlsanity.com/paper/2203.07559/) (2022)
- [Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity](https://mlsanity.com/paper/2210.15452/) (2022)
- [Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance](https://mlsanity.com/paper/2407.07950/) (2024)
- [Teaching LLMs to Abstain across Languages via Multilingual Feedback](https://mlsanity.com/paper/2406.15948/) (2024)
- [Calibration of Machine Reading Systems at Scale](https://mlsanity.com/paper/2203.10623/) (2022)
- [Humans overrely on overconfident language models, across languages](https://mlsanity.com/paper/2507.06306/) (2025)
- [A Formalism and Approach for Improving Robustness of Large Language Models Using Risk-Adjusted Confidence Scores](https://mlsanity.com/paper/2310.03283/) (2023)
- [Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets](https://mlsanity.com/paper/2109.07494/) (2021)
- [Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models](https://mlsanity.com/paper/2307.06713/) (2023)
- [Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inference](https://mlsanity.com/paper/2408.01935/) (2024)
- [Reliable Financial Named Entity Recognition Under Domain Shift: Confidence Estimation and Selective Prediction](https://mlsanity.com/paper/2608.19558/) (2026)
- [On-Device Named-Entity Recognition: A Deployability Study of Accuracy, Cost, Reliability, and Confidence](https://mlsanity.com/paper/2610.00007/) (2026)

## Abstract (arXiv)

We address the problem of calibrating prediction confidence for output entities of interest in natural language processing (NLP) applications. It is important that NLP applications such as named entity recognition and question answering produce calibrated confidence scores for their predictions, especially if the system is to be deployed in a safety-critical domain such as healthcare. However, the output space of such structured prediction models is often too large to adapt binary or multi-class calibration methods directly. In this study, we propose a general calibration scheme for output entities of interest in neural-network based structured prediction models. Our proposed method can be used with any binary class calibration scheme and a neural network model. Additionally, we show that our calibration method can also be used as an uncertainty-aware, entity-specific decoding step to improve the performance of the underlying model at no additional training cost or data requirements. We show that our method outperforms current calibration techniques for named-entity-recognition, part-of-speech and question answering. We also improve our model's performance from our decoding step across several tasks and benchmark datasets. Our method improves the calibration and model performance on out-of-domain test scenarios as well.