Learning Robust Kernel Ensembles with Kernel Average Pooling
Not on the map. Why?
Abstract (arXiv)
Model ensembles have long been used in machine learning to reduce the variance in individual model predictions, making them more robust to input perturbations. Pseudo-ensemble methods like dropout have also been commonly used in deep learning models to improve generalization. However, the application of these techniques to improve neural networks' robustness against input perturbations remains underexplored. We introduce Kernel Average Pooling (KAP), a neural network building block that applies the mean filter along the kernel dimension of the layer activation tensor. We show that ensembles of kernels with similar functionality naturally emerge in convolutional neural networks equipped with KAP and trained with backpropagation. Moreover, we show that when trained on inputs perturbed with additive Gaussian noise, KAP models are remarkably robust against various forms of adversarial attacks. Empirical evaluations on CIFAR10, CIFAR100, TinyImagenet, and Imagenet datasets show substantial improvements in robustness against strong adversarial attacks such as AutoAttack without training on any adversarial examples.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 50 of its 65 references that are in the map. First posted on arXiv 30 Sep 2022.
Builds on
50 of 65 references.
References · 50
- 2015Deep Residual Learning for Image Recognitioncited by 51,950
- 2014Explaining and Harnessing Adversarial Examplescited by 7,336
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
- 2013Intriguing properties of neural networkscited by 5,137
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,992
- 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,511
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,445
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,384
- 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,336
- 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,316
- 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,265
38 more
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,100
- 2017Ensemble Adversarial Training: Attacks and Defensescited by 839
- 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 738
- 2020Fast is better than free: Revisiting adversarial trainingcited by 725
- 2017Evasion Attacks against Machine Learning at Test Timecited by 689
- 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 684
- 2019Adversarial Training for Free!cited by 641
- 2020On Adaptive Attacks to Adversarial Example Defensescited by 554
- 2020Overfitting in adversarially robust deep learningcited by 505
- 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 467
- 2017Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Modelscited by 460
- 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
- 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 318
- 2018Feature Denoising for Improving Adversarial Robustnesscited by 313
- 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
- 2014Learning with Pseudo-Ensemblescited by 241
- 2021Improving Robustness using Generated Datacited by 234
- 2018Adversarial Logit Pairingcited by 224
- 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
- 2021Data Augmentation Can Improve Robustnesscited by 193
- 2013Maxout Networkscited by 186
- 2017Towards Robust Neural Networks via Random Self-ensemblecited by 158
- 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 124
- 2019Adversarial Examples Are a Natural Consequence of Test Error in Noisecited by 114
- 2021Robust Learning Meets Generative Models: Can Proxy Distributions Improve Adversarial Robustness?cited by 113
- 2020Boosting Adversarial Training with Hypersphere Embeddingcited by 97
- 2021Improving Adversarial Robustness via Channel-wise Activation Suppressingcited by 76
- 2019Improving Adversarial Robustness of Ensembles with Diversity Trainingcited by 69
- 2021Boosting Randomized Smoothing with Variance Reduced Classifierscited by 47
- 2019An Alternative Surrogate Loss for PGD-based Adversarial Testingcited by 45
- 2021Adversarial Robustness with Semi-Infinite Constrained Learningcited by 39
- 2021Attacking Adversarial Attacks as A Defensecited by 31
- 2021CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selectioncited by 21
- 2018Defensive Dropout for Hardening Deep Neural Networks under Adversarial Attackscited by 13
- 2020Adversarial Feature Desensitizationcited by 12
- 2021Neural Population Geometry Reveals the Role of Stochasticity in Robust Perceptioncited by 12
- 2020Toward Adversarial Robustness by Diversity in an Ensemble of Specialized Deep Neural Networkscited by 7
- 2021Get Fooled for the Right Reason: Improving Adversarial Robustness through a Teacher-guided Curriculum Learning Approachcited by 4
Built on by
Citing papers per year: 2024: 1.
1 paper.