# On the Certified Robustness for Ensemble Models and Beyond

arXiv: https://arxiv.org/abs/2107.10873 · HTML: https://mlsanity.com/paper/2107.10873/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 70 references that are in the map.
- First posted on arXiv 22 Jul 2021.

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [Certified Robustness to Adversarial Examples with Differential Privacy](https://mlsanity.com/paper/1802.03471/) (2018)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [Efficient Neural Network Robustness Certification with General Activation Functions](https://mlsanity.com/paper/1811.00866/) (2018)
- [Unlabeled Data Improves Adversarial Robustness](https://mlsanity.com/paper/1905.13736/) (2019)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers](https://mlsanity.com/paper/1906.04584/) (2019)
- [Characterizing Adversarial Subspaces Using Local Intrinsic Dimensionality](https://mlsanity.com/paper/1801.02613/) (2018)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [Towards Fast Computation of Certified Robustness for ReLU Networks](https://mlsanity.com/paper/1804.09699/) (2018)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [Spatially Transformed Adversarial Examples](https://mlsanity.com/paper/1801.02612/) (2018)
- [Certified Adversarial Robustness with Additive Noise](https://mlsanity.com/paper/1809.03113/) (2018)
- [Randomized Smoothing of All Shapes and Sizes](https://mlsanity.com/paper/2002.08118/) (2020)
- [MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius](https://mlsanity.com/paper/2001.02378/) (2020)
- [SoK: Certified Robustness for Deep Neural Networks](https://mlsanity.com/paper/2009.04131/) (2020)
- [Unrestricted Adversarial Examples via Semantic Manipulation](https://mlsanity.com/paper/1904.06347/) (2019)
- [Consistency Regularization for Certified Robustness of Smoothed Classifiers](https://mlsanity.com/paper/2006.04062/) (2020)
- [DVERGE: Diversifying Vulnerabilities for Enhanced Robust Generation of Ensembles](https://mlsanity.com/paper/2009.14720/) (2020)
- [Improving Adversarial Robustness of Ensembles with Diversity Training](https://mlsanity.com/paper/1901.09981/) (2019)

## Built on by (most cited first, 30 of 40)

- [(Certified!!) Adversarial Robustness for Free!](https://mlsanity.com/paper/2206.10550/) (2022)
- [LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustness](https://mlsanity.com/paper/2210.11620/) (2022)
- [Boosting the Certified Robustness of L-infinity Distance Nets](https://mlsanity.com/paper/2110.06850/) (2021)
- [Double Sampling Randomized Smoothing](https://mlsanity.com/paper/2206.07912/) (2022)
- [DiffSmooth: Certifiably Robust Learning via Diffusion Models and Local Smoothing](https://mlsanity.com/paper/2308.14333/) (2023)
- [MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers](https://mlsanity.com/paper/2402.02263/) (2024)
- [Quantifying and Enhancing Multi-modal Robustness with Modality Preference](https://mlsanity.com/paper/2402.06244/) (2024)
- [CARE: Certifiably Robust Learning with Reasoning via Variational Inference](https://mlsanity.com/paper/2209.05055/) (2022)
- [Verification of Neural Networks' Global Robustness](https://mlsanity.com/paper/2402.19322/) (2024)
- [Multi-scale Diffusion Denoised Smoothing](https://mlsanity.com/paper/2310.16779/) (2023)
- [Building Robust Ensembles via Margin Boosting](https://mlsanity.com/paper/2206.03362/) (2022)
- [(De-)Randomized Smoothing for Decision Stump Ensembles](https://mlsanity.com/paper/2205.13909/) (2022)
- [Data Models for Dataset Drift Controls in Machine Learning With Optical Images](https://mlsanity.com/paper/2211.02578/) (2022)
- [It's Simplex! Disaggregating Measures to Improve Certified Robustness](https://mlsanity.com/paper/2309.11005/) (2023)
- [A Recipe for Improved Certifiable Robustness](https://mlsanity.com/paper/2310.02513/) (2023)
- [SPLITZ: Certifiable Robustness via Split Lipschitz Randomized Smoothing](https://mlsanity.com/paper/2407.02811/) (2024)
- [Verifiable Learning for Robust Tree Ensembles](https://mlsanity.com/paper/2305.03626/) (2023)
- [Multi-head Ensemble of Smoothed Classifiers for Certified Robustness](https://mlsanity.com/paper/2211.10882/) (2022)
- [Certifying Ensembles: A General Certification Theory with S-Lipschitzness](https://mlsanity.com/paper/2304.13019/) (2023)
- [Consistent Explanations in the Face of Model Indeterminacy via Ensembling](https://mlsanity.com/paper/2306.06193/) (2023)
- [Effective Segmentation of Post-Treatment Gliomas Using Simple Approaches: Artificial Sequence Generation and Ensemble Models](https://mlsanity.com/paper/2409.08143/) (2024)
- [LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models](https://mlsanity.com/paper/2601.18513/) (2026)
- [How Sampling Impacts the Robustness of Stochastic Neural Networks](https://mlsanity.com/paper/2204.10839/) (2022)
- [Smooth-Reduce: Leveraging Patches for Improved Certified Robustness](https://mlsanity.com/paper/2205.06154/) (2022)
- [On the Perils of Cascading Robust Classifiers](https://mlsanity.com/paper/2206.00278/) (2022)
- [Automating global landslide detection with heterogeneous ensemble deep-learning classification](https://mlsanity.com/paper/2310.05959/) (2023)
- [Exploring Transferability for Randomized Smoothing](https://mlsanity.com/paper/2312.09020/) (2023)
- [Certified PEFTSmoothing: Parameter-Efficient Fine-Tuning with Randomized Smoothing](https://mlsanity.com/paper/2404.05350/) (2024)
- [Average Certified Radius is a Poor Metric for Randomized Smoothing](https://mlsanity.com/paper/2410.06895/) (2024)
- [On the Adversarial Transferability of Generalized "Skip Connections"](https://mlsanity.com/paper/2410.08950/) (2024)

## Abstract (arXiv)

Recent studies show that deep neural networks (DNN) are vulnerable to adversarial examples, which aim to mislead DNNs by adding perturbations with small magnitude. To defend against such attacks, both empirical and theoretical defense approaches have been extensively studied for a single ML model. In this work, we aim to analyze and provide the certified robustness for ensemble ML models, together with the sufficient and necessary conditions of robustness for different ensemble protocols. Although ensemble models are shown more robust than a single model empirically; surprisingly, we find that in terms of the certified robustness the standard ensemble models only achieve marginal improvement compared to a single model. Thus, to explore the conditions that guarantee to provide certifiably robust ensemble ML models, we first prove that diversified gradient and large confidence margin are sufficient and necessary conditions for certifiably robust ensemble models under the model-smoothness assumption. We then provide the bounded model-smoothness analysis based on the proposed Ensemble-before-Smoothing strategy. We also prove that an ensemble model can always achieve higher certified robustness than a single base model under mild conditions. Inspired by the theoretical findings, we propose the lightweight Diversity Regularized Training (DRT) to train certifiably robust ensemble ML models. Extensive experiments show that our DRT enhanced ensembles can consistently achieve higher certified robustness than existing single and ensemble ML models, demonstrating the state-of-the-art certified L2-robustness on MNIST, CIFAR-10, and ImageNet datasets.