Balancing Robustness and Sensitivity using Feature Contrastive Learning
Not on the map. Why?
Abstract (arXiv)
It is generally believed that robust training of extremely large networks is critical to their success in real-world applications. However, when taken to the extreme, methods that promote robustness can hurt the model's sensitivity to rare or underrepresented patterns. In this paper, we discuss this trade-off between sensitivity and robustness to natural (non-adversarial) perturbations by introducing two notions: contextual feature utility and contextual feature sensitivity. We propose Feature Contrastive Learning (FCL) that encourages a model to be more sensitive to the features that have higher contextual utility. Empirical results demonstrate that models trained with FCL achieve a better balance of robustness and sensitivity, leading to improved generalization in the presence of noise on both vision and NLP datasets.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 40 of its 49 references that are in the map. First posted on arXiv 19 May 2021.
Builds on
40 of 49 references.
References · 40
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2014Going Deeper with Convolutionscited by 4,231
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
28 more
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
- 2013Visualizing and Understanding Convolutional Networkscited by 2,278
- 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,510
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
- 2019Contrastive Multiview Codingcited by 1,297
- 2017SmoothGrad: removing noise by adding noisecited by 1,097
- 2018Robustness May Be at Odds with Accuracycited by 873
- 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
- 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
- 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 416
- 2018Adversarially Robust Generalization Requires More Datacited by 318
- 2019On the (In)fidelity and Sensitivity for Explanationscited by 309
- 2019Adversarial Examples Improve Image Recognitioncited by 283
- 2017Real Time Image Saliency for Black Box Classifierscited by 246
- 2019Towards Stable and Efficient Training of Verifiably Robust Neural Networkscited by 204
- 2017Visualizing Deep Neural Network Decisions: Prediction Difference Analysiscited by 166
- 2016Improving the Robustness of Deep Neural Networks via Stability Trainingcited by 156
- 2020Robust Pre-Training by Adversarial Contrastive Learningcited by 144
- 2020Adversarial Self-Supervised Contrastive Learningcited by 138
- 2019Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentationcited by 105
- 2018Improving DNN Robustness to Adversarial Attacks using Jacobian Regularizationcited by 88
- 2020Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbationscited by 55
- 2017Towards better understanding of gradient-based attribution methods for Deep Neural Networkscited by 51
- 2018Bilateral Adversarial Training: Towards Fast Training of More Robust Models Against Adversarial Attackscited by 48
- 2016Gradients of Counterfactualscited by 31
- 2019Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?cited by 25
- 2020Adversarial and Natural Perturbations for General Robustnesscited by 2
Built on by
Citing papers per year: 2023: 1.
1 paper.