# Foolbox: A Python toolbox to benchmark the robustness of machine learning models

arXiv: https://arxiv.org/abs/1707.04131 · HTML: https://mlsanity.com/paper/1707.04131/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2017.

## Built on by (most cited first, 20 of 20)

- [A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking](https://mlsanity.com/paper/2302.14301/) (2023)
- [Demystifying the Transferability of Adversarial Attacks in Computer Networks](https://mlsanity.com/paper/2110.04488/) (2021)
- [An Orthogonal Classifier for Improving the Adversarial Robustness of Neural Networks](https://mlsanity.com/paper/2105.09109/) (2021)
- [ROOM: Adversarial Machine Learning Attacks Under Real-Time Constraints](https://mlsanity.com/paper/2201.01621/) (2022)
- [Software Testing for Machine Learning](https://mlsanity.com/paper/2205.00210/) (2022)
- [Resisting Deep Learning Models Against Adversarial Attack Transferability via Feature Randomization](https://mlsanity.com/paper/2209.04930/) (2022)
- [Impeding LLM-assisted Cheating in Introductory Programming Assignments via Adversarial Perturbation](https://mlsanity.com/paper/2410.09318/) (2024)
- [SEC4SR: A Security Analysis Platform for Speaker Recognition](https://mlsanity.com/paper/2109.01766/) (2021)
- [Robust Sensible Adversarial Learning of Deep Neural Networks for Image Classification](https://mlsanity.com/paper/2205.10457/) (2022)
- [Fortify Machine Learning Production Systems: Detect and Classify Adversarial Attacks](https://mlsanity.com/paper/2102.09695/) (2021)
- [Defensive Tensorization](https://mlsanity.com/paper/2110.13859/) (2021)
- [From Environmental Sound Representation to Robustness of 2D CNN Models Against Adversarial Attacks](https://mlsanity.com/paper/2204.07018/) (2022)
- [Trace and Detect Adversarial Attacks on CNNs using Feature Response Maps](https://mlsanity.com/paper/2208.11436/) (2022)
- [CARE: Ensemble Adversarial Robustness Evaluation Against Adaptive Attackers for Security Applications](https://mlsanity.com/paper/2401.11126/) (2024)
- [PRBench: A Standardized Probabilistic Robustness Benchmark](https://mlsanity.com/paper/2511.01724/) (2025)
- [Understanding Adversarial Robustness from Feature Maps of Convolutional Layers](https://mlsanity.com/paper/2202.12435/) (2022)
- [Attacking Compressed Vision Transformers](https://mlsanity.com/paper/2209.13785/) (2022)
- [AccelAT: A Framework for Accelerating the Adversarial Training of Deep Neural Networks through Accuracy Gradient](https://mlsanity.com/paper/2210.06888/) (2022)
- [Unscrambling the Rectification of Adversarial Attacks Transferability across Computer Networks](https://mlsanity.com/paper/2311.03373/) (2023)
- [VCR-Bench: A Modular Open-Source Benchmark for Video Classification Robustness](https://mlsanity.com/paper/2610.08936/) (2026)

## Abstract (arXiv)

Even todays most advanced machine learning models are easily fooled by almost imperceptible perturbations of their inputs. Foolbox is a new Python package to generate such adversarial perturbations and to quantify and compare the robustness of machine learning models. It is build around the idea that the most comparable robustness measure is the minimum perturbation needed to craft an adversarial example. To this end, Foolbox provides reference implementations of most published adversarial attack methods alongside some new ones, all of which perform internal hyperparameter tuning to find the minimum adversarial perturbation. Additionally, Foolbox interfaces with most popular deep learning frameworks such as PyTorch, Keras, TensorFlow, Theano and MXNet and allows different adversarial criteria such as targeted misclassification and top-k misclassification as well as different distance measures. The code is licensed under the MIT license and is openly available at https://github.com/bethgelab/foolbox . The most up-to-date documentation can be found at http://foolbox.readthedocs.io .