mlsanityData to 14 Oct 2026

Confidence Calibration of Classifiers with Many Classes

arXiv 2411.02988 ↗First posted 5 Nov 2024cs.LG cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

For classification models based on neural networks, the maximum predicted class probability is often used as a confidence score. This score rarely predicts well the probability of making a correct prediction and requires a post-processing calibration step. However, many confidence calibration methods fail for problems with many classes. To address this issue, we transform the problem of calibrating a multiclass classifier into calibrating a single surrogate binary classifier. This approach allows for more efficient use of standard calibration methods. We evaluate our approach on numerous neural networks used for image or text classification and show that it significantly enhances existing calibration methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 55 of its 77 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

55 of 77 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  10. 2017On Calibration of Modern Neural Networkscited by 4,960
  11. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  12. 2022A ConvNet for the 2020scited by 3,579
38 more
  1. 2016Wide Residual Networkscited by 3,070
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  3. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  6. 2019Searching for MobileNetV3cited by 1,994
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  8. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  9. 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
  10. 2017Selective Classification for Deep Neural Networkscited by 852
  11. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  12. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  13. 2021ImageNet-21K Pretraining for the Massescited by 613
  14. 2019Measuring Calibration in Deep Learningcited by 446
  15. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  16. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  17. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  18. 2019Verified Uncertainty Calibrationcited by 309
  19. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  20. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  21. 2019Evaluating model calibration in classificationcited by 168
  22. 2020Bayesian Neural Networks: An Introduction and Surveycited by 131
  23. 2013From Amateurs to Connoisseurs: Modeling the Evolution of User Expertise through Online Reviewscited by 108
  24. 2020Multivariate Confidence Calibration for Object Detectioncited by 98
  25. 2020Calibration of Neural Networks using Splinescited by 97
  26. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  27. 2021Soft Calibration Objectives for Neural Networkscited by 92
  28. 2019Calibration tests in multi-class classification: A unifying frameworkcited by 79
  29. 2021Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibrationcited by 79
  30. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 75
  31. 2021Classifier Calibration: A survey on how to assess and improve predicted class probabilitiescited by 73
  32. 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
  33. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  34. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  35. 2021Top-label calibration and multiclass-to-binary reductionscited by 46
  36. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  37. 2020Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binningcited by 27
  38. 2023Generative Calibration for In-context Learningcited by 20

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
  2. 2025What Does It Take to Build a Performant Selective Classifier?cited by 2
  3. 2025Multiclass Local Calibration with the Jensen-Shannon Distancecited by 2
  4. 2025Adaptive Cumulative Mass Calibration with Conformal Predictioncited by 1
  5. 2025Probability Bounding: Post-Hoc Calibration via Box-Constrained Softmaxcited by 1
  6. 2025Newton-Puiseux Analysis for Interpretability and Calibration of Complex-Valued Neural Networkscited by 0
  7. 2026Multiclass Calibration Assessment and Recalibration of Probability Predictions via the Linear Log Odds Calibration Functioncited by 0
  8. 2026Socrates Loss: Unifying Confidence Calibration and Classification by Leveraging the Unknowncited by 0
  9. 2026Beyond Output-Space Calibration: Spectral Evidence Bundling for Selective Reliability Estimation in Time-Series Classificationcited by 0
  10. 2026Let Confidence Change, Not the Prediction: Prediction-Preserving Repair for Post-hoc Calibrationcited by 0
  11. 2026Signal-Routed Temperature Scaling: Low-Capacity Risk-Conditioned Calibration for Small Validation Budgetscited by 0