# TA3: Testing Against Adversarial Attacks on Machine Learning Models

arXiv: https://arxiv.org/abs/2410.05334 · HTML: https://mlsanity.com/paper/2410.05334/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 92 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Builds on (most cited first, 30 of 35)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Improving Transferability of Adversarial Examples with Input Diversity](https://mlsanity.com/paper/1803.06978/) (2018)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Recent Advances in Adversarial Training for Adversarial Robustness](https://mlsanity.com/paper/2102.01356/) (2021)
- [Adversarial Attacks and Defences: A Survey](https://mlsanity.com/paper/1810.00069/) (2018)
- [Dataset Distillation](https://mlsanity.com/paper/1811.10959/) (2018)
- [Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making](https://mlsanity.com/paper/1902.02960/) (2019)
- [ActiVis: Visual Exploration of Industry-Scale Deep Neural Network Models](https://mlsanity.com/paper/1704.01942/) (2017)
- ["Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practice](https://mlsanity.com/paper/2212.14315/) (2022)
- [Towards Better Analysis of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.07043/) (2016)
- [Universal Adversarial Attack on Attention and the Resulting Dataset DAmageNet](https://mlsanity.com/paper/2001.06325/) (2020)
- [Human-Centered Artificial Intelligence and Machine Learning](https://mlsanity.com/paper/1901.11184/) (2019)
- [A Survey of Black-Box Adversarial Attacks on Computer Vision Models](https://mlsanity.com/paper/1912.01667/) (2019)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [A Workflow for Visual Diagnostics of Binary Classifiers using Instance-Level Explanations](https://mlsanity.com/paper/1705.01968/) (2017)
- [A multi-component framework for the analysis and design of explainable artificial intelligence](https://mlsanity.com/paper/2005.01908/) (2020)
- [Evading Black-box Classifiers Without Breaking Eggs](https://mlsanity.com/paper/2306.02895/) (2023)
- [Towards Involving End-users in Interactive Human-in-the-loop AI Fairness](https://mlsanity.com/paper/2204.10464/) (2022)
- [Debiased-CAM to mitigate image perturbations with faithful visual explanations of machine learning](https://mlsanity.com/paper/2012.05567/) (2020)
- [EDAssistant: Supporting Exploratory Data Analysis in Computational Notebooks with In-Situ Code Search and Recommendation](https://mlsanity.com/paper/2112.07858/) (2021)
- [Integrated Visualization Editing via Parameterized Declarative Templates](https://mlsanity.com/paper/2101.07902/) (2021)
- [Explainable Global Fairness Verification of Tree-Based Classifiers](https://mlsanity.com/paper/2209.13179/) (2022)
- [Segment (Almost) Nothing: Prompt-Agnostic Adversarial Attacks on Segmentation Models](https://mlsanity.com/paper/2311.14450/) (2023)

## Abstract (arXiv)

Adversarial attacks are major threats to the deployment of machine learning (ML) models in many applications. Testing ML models against such attacks is becoming an essential step for evaluating and improving ML models. In this paper, we report the design and development of an interactive system for aiding the workflow of Testing Against Adversarial Attacks (TA3). In particular, with TA3, human-in-the-loop (HITL) enables human-steered attack simulation and visualization-assisted attack impact evaluation. While the current version of TA3 focuses on testing decision tree models against adversarial attacks based on the One Pixel Attack Method, it demonstrates the importance of HITL in ML testing and the potential application of HITL to the ML testing workflows for other types of ML models and other types of adversarial attacks.