# URET: Universal Robustness Evaluation Toolkit (for Evasion)

arXiv: https://arxiv.org/abs/2308.01840 · HTML: https://mlsanity.com/paper/2308.01840/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Builds on (most cited first, 30 of 33)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [TextBugger: Generating Adversarial Text Against Real-world Applications](https://mlsanity.com/paper/1812.05271/) (2018)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [Towards Stable and Efficient Training of Verifiably Robust Neural Networks](https://mlsanity.com/paper/1906.06316/) (2019)
- [EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models](https://mlsanity.com/paper/1804.04637/) (2018)
- [ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector](https://mlsanity.com/paper/1804.05810/) (2018)
- [Technical Report on the CleverHans v2.1.0 Adversarial Examples Library](https://mlsanity.com/paper/1610.00768/) (2016)
- [Adversarial Machine Learning -- Industry Perspectives](https://mlsanity.com/paper/2002.05646/) (2020)
- [Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks](https://mlsanity.com/paper/2106.09249/) (2021)
- [Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN](https://mlsanity.com/paper/1702.05983/) (2017)
- [advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch](https://mlsanity.com/paper/1902.07623/) (2019)
- [Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learning](https://mlsanity.com/paper/1801.08917/) (2018)
- [Exploring Adversarial Examples in Malware Detection](https://mlsanity.com/paper/1810.08280/) (2018)
- [AugLy: Data Augmentations for Robustness](https://mlsanity.com/paper/2201.06494/) (2022)
- [Invisible Mask: Practical Attacks on Face Recognition with Infrared](https://mlsanity.com/paper/1803.04683/) (2018)
- [Adversarial Deep Learning for Robust Detection of Binary Encoded Malware](https://mlsanity.com/paper/1801.02950/) (2018)
- [Malware Makeover: Breaking ML-based Static Analysis by Modifying Executable Bytes](https://mlsanity.com/paper/1912.09064/) (2019)
- [Advbox: a toolbox to generate adversarial examples that fool neural networks](https://mlsanity.com/paper/2001.05574/) (2020)

## Built on by (most cited first, 3 of 3)

- [Systematically Assessing the Security Risks of AI/ML-enabled Connected Healthcare Systems](https://mlsanity.com/paper/2401.17136/) (2024)
- [On the Security Risks of ML-based Malware Detection Systems: A Survey](https://mlsanity.com/paper/2505.10903/) (2025)
- [BotSSCL: Social Bot Detection with Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2402.03740/) (2024)

## Abstract (arXiv)

Machine learning models are known to be vulnerable to adversarial evasion attacks as illustrated by image classification models. Thoroughly understanding such attacks is critical in order to ensure the safety and robustness of critical AI tasks. However, most evasion attacks are difficult to deploy against a majority of AI systems because they have focused on image domain with only few constraints. An image is composed of homogeneous, numerical, continuous, and independent features, unlike many other input types to AI systems used in practice. Furthermore, some input types include additional semantic and functional constraints that must be observed to generate realistic adversarial inputs. In this work, we propose a new framework to enable the generation of adversarial inputs irrespective of the input type and task domain. Given an input and a set of pre-defined input transformations, our framework discovers a sequence of transformations that result in a semantically correct and functional adversarial input. We demonstrate the generality of our approach on several diverse machine learning tasks with various input representations. We also show the importance of generating adversarial examples as they enable the deployment of mitigation techniques.