# Robust Sparse Regularization: Simultaneously Optimizing Neural Network Robustness and Compactness

arXiv: https://arxiv.org/abs/1905.13074 · HTML: https://mlsanity.com/paper/1905.13074/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2019.

## Built on by (most cited first, 7 of 7)

- [Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks](https://mlsanity.com/paper/2110.14068/) (2021)
- [MIA-Former: Efficient and Robust Vision Transformers via Multi-grained Input-Adaptation](https://mlsanity.com/paper/2112.11542/) (2021)
- [Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness](https://mlsanity.com/paper/2409.01249/) (2024)
- [Balance is Essence: Accelerating Sparse Training via Adaptive Gradient Correction](https://mlsanity.com/paper/2301.03573/) (2023)
- [CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial Robustness](https://mlsanity.com/paper/2212.01957/) (2022)
- [Fragility, Robustness and Antifragility in Deep Learning](https://mlsanity.com/paper/2312.09821/) (2023)
- [Analytic Continuation by Feature Learning](https://mlsanity.com/paper/2411.17728/) (2024)

## Abstract (arXiv)

Deep Neural Network (DNN) trained by the gradient descent method is known to be vulnerable to maliciously perturbed adversarial input, aka. adversarial attack. As one of the countermeasures against adversarial attack, increasing the model capacity for DNN robustness enhancement was discussed and reported as an effective approach by many recent works. In this work, we show that shrinking the model size through proper weight pruning can even be helpful to improve the DNN robustness under adversarial attack. For obtaining a simultaneously robust and compact DNN model, we propose a multi-objective training method called Robust Sparse Regularization (RSR), through the fusion of various regularization techniques, including channel-wise noise injection, lasso weight penalty, and adversarial training. We conduct extensive experiments across popular ResNet-20, ResNet-18 and VGG-16 DNN architectures to demonstrate the effectiveness of RSR against popular white-box (i.e., PGD and FGSM) and black-box attacks. Thanks to RSR, 85% weight connections of ResNet-18 can be pruned while still achieving 0.68% and 8.72% improvement in clean- and perturbed-data accuracy respectively on CIFAR-10 dataset, in comparison to its PGD adversarial training baseline.