# Robust Few-shot Learning Without Using any Adversarial Samples

arXiv: https://arxiv.org/abs/2211.01598 · HTML: https://mlsanity.com/paper/2211.01598/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 3 Nov 2022.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Meta-learning with differentiable closed-form solvers](https://mlsanity.com/paper/1805.08136/) (2018)
- [High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks](https://mlsanity.com/paper/1905.13545/) (2019)
- [A Baseline for Few-Shot Image Classification](https://mlsanity.com/paper/1909.02729/) (2019)
- [Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning](https://mlsanity.com/paper/2012.09816/) (2020)
- [Are Labels Required for Improving Adversarial Robustness?](https://mlsanity.com/paper/1905.13725/) (2019)
- [Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better](https://mlsanity.com/paper/2108.07969/) (2021)
- [Jacobian Adversarially Regularized Networks for Robustness](https://mlsanity.com/paper/1912.10185/) (2019)
- [Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong](https://mlsanity.com/paper/1706.04701/) (2017)
- [Towards Frequency-Based Explanation for Robust CNN](https://mlsanity.com/paper/2005.03141/) (2020)
- [Adversarial Meta-Learning](https://mlsanity.com/paper/1806.03316/) (2018)
- [Improving Adversarial Robustness via Guided Complement Entropy](https://mlsanity.com/paper/1903.09799/) (2019)
- [Towards Achieving Adversarial Robustness by Enforcing Feature Consistency Across Bit Planes](https://mlsanity.com/paper/2004.00306/) (2020)
- [Mimic and Fool: A Task Agnostic Adversarial Attack](https://mlsanity.com/paper/1906.04606/) (2019)
- [Finding Physical Adversarial Examples for Autonomous Driving with Fast and Differentiable Image Compositing](https://mlsanity.com/paper/2010.08844/) (2020)
- [InfoAT: Improving Adversarial Training Using the Information Bottleneck Principle](https://mlsanity.com/paper/2206.12292/) (2022)
- [Long-term Cross Adversarial Training: A Robust Meta-learning Method for Few-shot Classification Tasks](https://mlsanity.com/paper/2106.12900/) (2021)

## Abstract (arXiv)

The high cost of acquiring and annotating samples has made the `few-shot' learning problem of prime importance. Existing works mainly focus on improving performance on clean data and overlook robustness concerns on the data perturbed with adversarial noise. Recently, a few efforts have been made to combine the few-shot problem with the robustness objective using sophisticated Meta-Learning techniques. These methods rely on the generation of adversarial samples in every episode of training, which further adds a computational burden. To avoid such time-consuming and complicated procedures, we propose a simple but effective alternative that does not require any adversarial samples. Inspired by the cognitive decision-making process in humans, we enforce high-level feature matching between the base class data and their corresponding low-frequency samples in the pretraining stage via self distillation. The model is then fine-tuned on the samples of novel classes where we additionally improve the discriminability of low-frequency query set features via cosine similarity. On a 1-shot setting of the CIFAR-FS dataset, our method yields a massive improvement of 60.55% & 62.05% in adversarial accuracy on the PGD and state-of-the-art Auto Attack, respectively, with a minor drop in clean accuracy compared to the baseline. Moreover, our method only takes 1.69× of the standard training time while being ≈ 5× faster than state-of-the-art adversarial meta-learning methods. The code is available at https://github.com/vcl-iisc/robust-few-shot-learning.