# Two Heads are Actually Better than One: Towards Better Adversarial Robustness via Transduction and Rejection

arXiv: https://arxiv.org/abs/2305.17528 · HTML: https://mlsanity.com/paper/2305.17528/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 38 references that are in the map.
- First posted on arXiv 27 May 2023.

## Builds on (most cited first, 25 of 25)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [RobustBench: a standardized adversarial robustness benchmark](https://mlsanity.com/paper/2010.09670/) (2020)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [VC Classes are Adversarially Robustly Learnable, but Only Improperly](https://mlsanity.com/paper/1902.04217/) (2019)
- [Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them](https://mlsanity.com/paper/2107.11630/) (2021)
- [Sample compression schemes for VC classes](https://mlsanity.com/paper/1503.06960/) (2015)
- [Beyond Perturbations: Learning Guarantees with Arbitrary Adversarial Test Examples](https://mlsanity.com/paper/2007.05145/) (2020)
- [Deep Neural Rejection against Adversarial Examples](https://mlsanity.com/paper/1910.00470/) (2019)
- [Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks](https://mlsanity.com/paper/2105.08714/) (2021)
- [Sample Compression for Real-Valued Learners](https://mlsanity.com/paper/1805.08254/) (2018)
- [A Research Agenda: Dynamic Models to Defend Against Correlated Attacks](https://mlsanity.com/paper/1903.06293/) (2019)
- [Two Coupled Rejection Metrics Can Tell Adversarial Examples Apart](https://mlsanity.com/paper/2105.14785/) (2021)
- [Increasing Confidence in Adversarial Robustness Evaluations](https://mlsanity.com/paper/2206.13991/) (2022)
- [Transductive Robust Learning Guarantees](https://mlsanity.com/paper/2110.10602/) (2021)
- [Playing it Safe: Adversarial Robustness with an Abstain Option](https://mlsanity.com/paper/1911.11253/) (2019)
- [ATRO: Adversarial Training with a Rejection Option](https://mlsanity.com/paper/2010.12905/) (2020)
- [Towards Evaluating the Robustness of Neural Networks Learned by Transduction](https://mlsanity.com/paper/2110.14735/) (2021)
- [Be Your Own Neighborhood: Detecting Adversarial Example by the Neighborhood Relations Built on Self-Supervised Learning](https://mlsanity.com/paper/2209.00005/) (2022)
- [Improving Adversarial Robustness via Joint Classification and Multiple Explicit Detection Classes](https://mlsanity.com/paper/2210.14410/) (2022)

## Abstract (arXiv)

Both transduction and rejection have emerged as important techniques for defending against adversarial perturbations. A recent work by Goldwasser et al. showed that rejection combined with transduction can give provable guarantees (for certain problems) that cannot be achieved otherwise. Nevertheless, under recent strong adversarial attacks, their work was shown to have low performance in a practical deep-learning setting. In this paper, we take a step towards realizing the promise of transduction+rejection in more realistic scenarios. Our key observation is that a novel application of a reduction technique by Tram`er, which was until now only used to demonstrate the vulnerability of certain defenses, can be used to actually construct effective defenses. Theoretically, we show that a careful application of this technique in the transductive setting can give significantly improved sample-complexity for robust generalization. Our theory guides us to design a new transductive algorithm for learning a selective model; extensive experiments using state of the art attacks show that our approach provides significantly better robust accuracy (81.6% on CIFAR-10 and 57.9% on CIFAR-100 under l_∞ with budget 8/255) than existing techniques.