# Stochastic Local Winner-Takes-All Networks Enable Profound Adversarial Robustness

arXiv: https://arxiv.org/abs/2112.02671 · HTML: https://mlsanity.com/paper/2112.02671/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 5 Dec 2021.

## Builds on (most cited first, 17 of 17)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Overfitting in adversarially robust deep learning](https://mlsanity.com/paper/2002.11569/) (2020)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples](https://mlsanity.com/paper/2010.03593/) (2020)
- [Attacks Which Do Not Kill Training Make Adversarial Learning Stronger](https://mlsanity.com/paper/2002.11242/) (2020)
- [Boosting Adversarial Training with Hypersphere Embedding](https://mlsanity.com/paper/2002.08619/) (2020)
- [Do Wider Neural Networks Really Help Adversarial Robustness?](https://mlsanity.com/paper/2010.01279/) (2020)
- [The Limitations of Adversarial Training and the Blind-Spot Attack](https://mlsanity.com/paper/1901.04684/) (2019)
- [Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translation](https://mlsanity.com/paper/2109.13318/) (2021)
- [Nonparametric Bayesian Deep Networks with Local Competition](https://mlsanity.com/paper/1805.07624/) (2018)
- [Local Competition and Stochasticity for Adversarial Robustness in Deep Learning](https://mlsanity.com/paper/2101.01121/) (2021)

## Built on by (most cited first, 11 of 11)

- [Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling](https://mlsanity.com/paper/2205.14283/) (2022)
- [DISCOVER: Making Vision Networks Interpretable via Competition and Dissection](https://mlsanity.com/paper/2310.04929/) (2023)
- [Synaptic Plasticity Models and Bio-Inspired Unsupervised Deep Learning: A Survey](https://mlsanity.com/paper/2307.16236/) (2023)
- [A New Dataset for End-to-End Sign Language Translation: The Greek Elementary School Dataset](https://mlsanity.com/paper/2310.04753/) (2023)
- [Competing Mutual Information Constraints with Stochastic Competition-based Activations for Learning Diversified Representations](https://mlsanity.com/paper/2201.03624/) (2022)
- [On the Role of Randomization in Adversarially Robust Classification](https://mlsanity.com/paper/2302.07221/) (2023)
- [Causal Perception Inspired Representation Learning for Trustworthy Image Quality Assessment](https://mlsanity.com/paper/2404.19567/) (2024)
- [Enhancing Adversarial Robustness through Multi-Objective Representation Learning](https://mlsanity.com/paper/2410.01697/) (2024)
- [Lattice Climber Attack: Adversarial attacks for randomized mixtures of classifiers](https://mlsanity.com/paper/2506.10888/) (2025)
- [Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNs](https://mlsanity.com/paper/2508.14140/) (2025)
- [Colliding with Adversaries at ECML-PKDD 2025 Model Robustness Competition 1st Prize Solution](https://mlsanity.com/paper/2510.16443/) (2025)

## Abstract (arXiv)

This work explores the potency of stochastic competition-based activations, namely Stochastic Local Winner-Takes-All (LWTA), against powerful (gradient-based) white-box and black-box adversarial attacks; we especially focus on Adversarial Training settings. In our work, we replace the conventional ReLU-based nonlinearities with blocks comprising locally and stochastically competing linear units. The output of each network layer now yields a sparse output, depending on the outcome of winner sampling in each block. We rely on the Variational Bayesian framework for training and inference; we incorporate conventional PGD-based adversarial training arguments to increase the overall adversarial robustness. As we experimentally show, the arising networks yield state-of-the-art robustness against powerful adversarial attacks while retaining very high classification rate in the benign case.