mlsanityData to 7 Oct 2026

Calibration Attacks: A Comprehensive Study of Adversarial Attacks on Model Confidence

arXiv 2401.02718 ↗First posted 5 Jan 2024cs.LG cs.CRcited by 7

Not on the map. Why?

Abstract (arXiv)

In this work, we highlight and perform a comprehensive study on calibration attacks, a form of adversarial attacks that aim to trap victim models to be heavily miscalibrated without altering their predicted labels, hence endangering the trustworthiness of the models and follow-up decision making based on their confidence. We propose four typical forms of calibration attacks: underconfidence, overconfidence, maximum miscalibration, and random confidence attacks, conducted in both black-box and white-box setups. We demonstrate that the attacks are highly effective on both convolutional and attention-based models: with a small number of queries, they seriously skew confidence without changing the predictive performance. Given the potential danger, we further investigate the effectiveness of a wide range of adversarial defence and recalibration methods, including our proposed defences specifically designed for calibration attacks to mitigate the harm. From the ECE and KS scores, we observe that there are still significant limitations in handling calibration attacks. To the best of our knowledge, this is the first dedicated study that provides a comprehensive investigation on calibration-focused attacks. We hope this study helps attract more attention to these types of attacks and hence hamper their potential serious damages. To this end, this work also provides detailed analyses to understand the characteristics of the attacks. Our code is available at https://github.com/PhenetOs/CalibrationAttack

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 5 Jan 2024.

Builds on

50 of 79 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2016Wide Residual Networkscited by 3,057
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
38 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  4. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  5. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  6. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  7. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
  8. 2018Improving Transferability of Adversarial Examples with Input Diversitycited by 591
  9. 2019Measuring Calibration in Deep Learningcited by 443
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  12. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  13. 2019Verified Uncertainty Calibrationcited by 307
  14. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  15. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  16. 2018Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionalitycited by 280
  17. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 275
  18. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  19. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  20. 2021Data Augmentation Can Improve Robustnesscited by 193
  21. 2018Stochastic Activation Pruning for Robust Adversarial Defensecited by 171
  22. 2022Robustness and Accuracy Could Be Reconcilable by (Proper) Definitioncited by 109
  23. 2020Calibration of Neural Networks using Splinescited by 97
  24. 2020A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scorescited by 74
  25. 2021Boosting Adversarial Transferability through Enhanced Momentumcited by 53
  26. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39
  27. 2020Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classificationcited by 37
  28. 2020Certifying Confidence via Randomized Smoothingcited by 36
  29. 2020Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibrationcited by 34
  30. 2022DensePure: Understanding Diffusion Models towards Adversarial Robustnesscited by 33
  31. 2021SpectralDefense: Detecting Adversarial Attacks on CNNs in the Fourier Domaincited by 30
  32. 2020Improving Calibration through the Relationship with Adversarial Robustnesscited by 26
  33. 2023A Diachronic Perspective on User Trust in AI under Uncertaintycited by 23
  34. 2021Disrupting Deep Uncertainty Estimation Without Harming Accuracycited by 19
  35. 2019On-manifold Adversarial Data Augmentation Improves Uncertainty Calibrationcited by 18
  36. 2022Adversarial Attack on Attackers: Post-Process to Mitigate Black-Box Score-Based Query Attackscited by 18
  37. 2022Increasing Confidence in Adversarial Robustness Evaluationscited by 12
  38. 2019The Threat of Adversarial Attacks on Machine Learning in Network Security -- A Surveycited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 5.

7 papers.

  1. 2024SMARTCAL: An Approach to Self-Aware Tool-Use Evaluation and Calibrationcited by 13
  2. 2025On the Robustness of Verbal Confidence of LLMs in Adversarial Attackscited by 3
  3. 2026Temperature Scaling Attack Disrupting Model Confidence in Federated Learningcited by 0
  4. 2026The Confidence Trap: Calibration Attacks for Graph Neural Networkscited by 0
  5. 2026LiST: Lipschitz Scaling Training for Robust and Calibrated Neural Networkscited by 0
  6. 2026Vision-Language Model Confidence Is Not a Property of the Answercited by 0
  7. 2026Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Modelscited by 0