# Mental Models of Adversarial Machine Learning

arXiv: https://arxiv.org/abs/2105.03726 · HTML: https://mlsanity.com/paper/2105.03726/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 90 references that are in the map.
- First posted on arXiv 8 May 2021.

## Builds on (most cited first, 20 of 20)

- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning](https://mlsanity.com/paper/1712.05526/) (2017)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Machine Learning with Membership Privacy using Adversarial Regularization](https://mlsanity.com/paper/1807.05852/) (2018)
- [Adversarial Machine Learning -- Industry Perspectives](https://mlsanity.com/paper/2002.05646/) (2020)
- [What Do We Want From Explainable Artificial Intelligence (XAI)? -- A Stakeholder Perspective on XAI and a Conceptual Model Guiding Interdisciplinary XAI Research](https://mlsanity.com/paper/2102.07817/) (2021)
- [Adversarial Reprogramming of Neural Networks](https://mlsanity.com/paper/1806.11146/) (2018)
- [Sponge Examples: Energy-Latency Attacks on Neural Networks](https://mlsanity.com/paper/2006.03463/) (2020)
- [Bypassing Backdoor Detection Algorithms in Deep Learning](https://mlsanity.com/paper/1905.13409/) (2019)
- [How do Data Science Workers Collaborate? Roles, Workflows, and Tools](https://mlsanity.com/paper/2001.06684/) (2020)
- [Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs](https://mlsanity.com/paper/2101.09824/) (2021)
- [Motivating the Rules of the Game for Adversarial Example Research](https://mlsanity.com/paper/1807.06732/) (2018)
- [Model-Reuse Attacks on Deep Learning Systems](https://mlsanity.com/paper/1812.00483/) (2018)
- [Bad Global Minima Exist and SGD Can Reach Them](https://mlsanity.com/paper/1906.02613/) (2019)
- [On the Robustness of Convolutional Neural Networks to Internal Architecture and Weight Perturbations](https://mlsanity.com/paper/1703.08245/) (2017)
- [Security and Machine Learning in the Real World](https://mlsanity.com/paper/2007.07205/) (2020)
- [Understanding Mental Models of AI through Player-AI Interaction](https://mlsanity.com/paper/2103.16168/) (2021)
- [Towards Reverse-Engineering Black-Box Neural Networks](https://mlsanity.com/paper/1711.01768/) (2017)
- [Why Do Developers Get Password Storage Wrong? A Qualitative Usability Study](https://mlsanity.com/paper/1708.08759/) (2017)
- [Surveying the Landscape of Ethics-Focused Design Methods](https://mlsanity.com/paper/2102.08909/) (2021)

## Abstract (arXiv)

Although machine learning is widely used in practice, little is known about practitioners' understanding of potential security challenges. In this work, we close this substantial gap and contribute a qualitative study focusing on developers' mental models of the machine learning pipeline and potentially vulnerable components. Similar studies have helped in other security fields to discover root causes or improve risk communication. Our study reveals two facets of practitioners' mental models of machine learning security. Firstly, practitioners often confuse machine learning security with threats and defences that are not directly related to machine learning. Secondly, in contrast to most academic research, our participants perceive security of machine learning as not solely related to individual models, but rather in the context of entire workflows that consist of multiple components. Jointly with our additional findings, these two facets provide a foundation to substantiate mental models for machine learning security and have implications for the integration of adversarial machine learning into corporate workflows, newdecreasing practitioners' reported uncertainty, and appropriate regulatory frameworks for machine learning security.