# Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets

arXiv: https://arxiv.org/abs/2109.07494 · HTML: https://mlsanity.com/paper/2109.07494/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 15 Sep 2021.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning](https://mlsanity.com/paper/2006.13092/) (2020)
- [Comprehensive Supersense Disambiguation of English Prepositions and Possessives](https://mlsanity.com/paper/1805.04905/) (2018)
- [Calibrating Structured Output Predictors for Natural Language Processing](https://mlsanity.com/paper/2004.04361/) (2020)
- [Supertagging the Long Tail with Tree-Structured Decoding of Complex Categories](https://mlsanity.com/paper/2012.01285/) (2020)
- [Lexical Semantic Recognition](https://mlsanity.com/paper/2004.15008/) (2020)

## Built on by (most cited first, 1 of 1)

- [How to Estimate Whether You Have Found Several Needles in a Haystack: Measuring Calibration in Multi-Label Text Classification](https://mlsanity.com/paper/2609.26468/) (2026)

## Abstract (arXiv)

For interpreting the behavior of a probabilistic model, it is useful to measure a model's calibration--the extent to which it produces reliable confidence scores. We address the open problem of calibration for tagging models with sparse tagsets, and recommend strategies to measure and reduce calibration error (CE) in such models. We show that several post-hoc recalibration techniques all reduce calibration error across the marginal distribution for two existing sequence taggers. Moreover, we propose tag frequency grouping (TFG) as a way to measure calibration error in different frequency bands. Further, recalibrating each group separately promotes a more equitable reduction of calibration error across the tag frequency spectrum.