# Towards Compact and Robust Deep Neural Networks

arXiv: https://arxiv.org/abs/1906.06110 · HTML: https://mlsanity.com/paper/1906.06110/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2019.

## Built on by (most cited first, 23 of 23)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy](https://mlsanity.com/paper/2103.03014/) (2021)
- [Pruning has a disparate impact on model accuracy](https://mlsanity.com/paper/2205.13574/) (2022)
- [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022)
- [Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network](https://mlsanity.com/paper/2204.02738/) (2022)
- [Robust Lottery Tickets for Pre-trained Language Models](https://mlsanity.com/paper/2211.03013/) (2022)
- [Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free](https://mlsanity.com/paper/2205.11819/) (2022)
- [Intriguing Properties of Compression on Multilingual Models](https://mlsanity.com/paper/2211.02738/) (2022)
- [Linearity Grafting: Relaxed Neuron Pruning Helps Certifiable Robustness](https://mlsanity.com/paper/2206.07839/) (2022)
- [Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness](https://mlsanity.com/paper/2409.01249/) (2024)
- [Integer-arithmetic-only Certified Robustness for Quantized Neural Networks](https://mlsanity.com/paper/2108.09413/) (2021)
- [Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data](https://mlsanity.com/paper/2205.08358/) (2022)
- [A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models](https://mlsanity.com/paper/2210.05211/) (2022)
- [RIBAC: Towards Robust and Imperceptible Backdoor Attack against Compact DNN](https://mlsanity.com/paper/2208.10608/) (2022)
- [CSTAR: Towards Compact and STructured Deep Neural Networks with Adversarial Robustness](https://mlsanity.com/paper/2212.01957/) (2022)
- [Deadwooding: Robust Global Pruning for Deep Neural Networks](https://mlsanity.com/paper/2202.05226/) (2022)
- [Samples on Thin Ice: Re-Evaluating Adversarial Pruning of Neural Networks](https://mlsanity.com/paper/2310.08073/) (2023)
- [ARQ: A Mixed-Precision Quantization Framework for Accurate and Certifiably Robust DNNs](https://mlsanity.com/paper/2410.24214/) (2024)
- [VeriCompress: A Tool to Streamline the Synthesis of Verified Robust Compressed Neural Networks from Scratch](https://mlsanity.com/paper/2211.09945/) (2022)
- [CCSRP: Robust Pruning of Spiking Neural Networks through Cooperative Coevolution](https://mlsanity.com/paper/2408.00794/) (2024)
- [S2AP: Score-space Sharpness Minimization for Adversarial Pruning](https://mlsanity.com/paper/2510.18381/) (2025)
- [Bias In, Bias Out? Finding Unbiased Subnetworks in Vanilla Models](https://mlsanity.com/paper/2603.05582/) (2026)
- [Safety-Potential Pruning for Enhancing Safety Prompts Against VLM Jailbreaking Without Retraining](https://mlsanity.com/paper/2603.14219/) (2026)

## Abstract (arXiv)

Deep neural networks have achieved impressive performance in many applications but their large number of parameters lead to significant computational and storage overheads. Several recent works attempt to mitigate these overheads by designing compact networks using pruning of connections. However, we observe that most of the existing strategies to design compact networks fail to preserve network robustness against adversarial examples. In this work, we rigorously study the extension of network pruning strategies to preserve both benign accuracy and robustness of a network. Starting with a formal definition of the pruning procedure, including pre-training, weights pruning, and fine-tuning, we propose a new pruning method that can create compact networks while preserving both benign accuracy and robustness. Our method is based on two main insights: (1) we ensure that the training objectives of the pre-training and fine-tuning steps match the training objective of the desired robust model (e.g., adversarial robustness/verifiable robustness), and (2) we keep the pruning strategy agnostic to pre-training and fine-tuning objectives. We evaluate our method on four different networks on the CIFAR-10 dataset and measure benign accuracy, empirical robust accuracy, and verifiable robust accuracy. We demonstrate that our pruning method can preserve on average 93% benign accuracy, 92.5% empirical robust accuracy, and 85.0% verifiable robust accuracy while compressing the tested network by 10×.