Impact of Regularization on Calibration and Robustness: from the Representation Space Perspective
Not on the map. Why?
Abstract (arXiv)
Recent studies have shown that regularization techniques using soft labels, e.g., label smoothing, Mixup, and CutMix, not only enhance image classification accuracy but also mitigate miscalibration due to overconfident predictions, and improve robustness against adversarial attacks. However, the underlying mechanisms of such improvements remain underexplored. In this paper, we offer a novel explanation from the perspective of the representation space (i.e., the space of the features obtained at the penultimate layer). Based on examination of decision boundaries and structure of features (or representation vectors), our study investigates confidence contours and gradient directions within the representation space. Furthermore, we analyze the adjustments in feature distributions due to regularization in relation to these contours and directions, from which we uncover central mechanisms inducing improved calibration and robustness. Our findings provide new insights into the characteristics of the high-dimensional representation space in relation to training and regularization using soft labels.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 37 of its 41 references that are in the map. First posted on arXiv 5 Oct 2024.
Builds on
37 of 41 references.
References · 37
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
- 2013Intriguing properties of neural networkscited by 5,155
- 2017On Calibration of Modern Neural Networkscited by 4,960
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
25 more
- 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2022A ConvNet for the 2020scited by 3,579
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
- 2021Convex optimizationcited by 2,515
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
- 2019When Does Label Smoothing Help?cited by 988
- 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 737
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2021Revisiting the Calibration of Modern Neural Networkscited by 390
- 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
- 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
- 2017NormFace: L2 Hypersphere Embedding for Face Verificationcited by 237
- 2021Vision Transformers are Robust Learnerscited by 207
- 2020How Does Mixup Help With Robustness and Generalization?cited by 193
- 2021Are Transformers More Robust Than CNNs?cited by 187
- 2019A Strong Baseline and Batch Normalization Neck for Deep Person Re-identificationcited by 155
- 2021Adversarial Robustness Comparison of Vision Transformer and MLP-Mixer to CNNscited by 57
- 2023Understanding and Improving Ensemble Adversarial Defensecited by 14
- 2020Label Smoothing and Adversarial Robustnesscited by 10
- 2023On the Robustness of Neural Collapse and the Neural Collapse of Robustnesscited by 9
- 2022Curved Representation Space of Vision Transformerscited by 8
- 2024Pushing Boundaries: Mixup's Influence on Neural Collapsecited by 7
- 2024HyperSpaceX: Radial and Angular Exploration of HyperSpherical Dimensionscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.