# Evaluating the Cybersecurity Risk of Real World, Machine Learning Production Systems

arXiv: https://arxiv.org/abs/2107.01806 · HTML: https://mlsanity.com/paper/2107.01806/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 108 references that are in the map.
- First posted on arXiv 5 Jul 2021.

## Builds on (most cited first, 30 of 33)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [DeepFool: a simple and accurate method to fool deep neural networks](https://mlsanity.com/paper/1511.04599/) (2015)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models](https://mlsanity.com/paper/1708.03999/) (2017)
- [Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods](https://mlsanity.com/paper/1705.07263/) (2017)
- [Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models](https://mlsanity.com/paper/1712.04248/) (2017)
- [Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples](https://mlsanity.com/paper/1605.07277/) (2016)
- [Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning](https://mlsanity.com/paper/1712.03141/) (2017)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning](https://mlsanity.com/paper/1804.00308/) (2018)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [Information Leakage in Embedding Models](https://mlsanity.com/paper/2004.00053/) (2020)
- [Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization](https://mlsanity.com/paper/1708.08689/) (2017)
- [PRADA: Protecting against DNN Model Stealing Attacks](https://mlsanity.com/paper/1805.02628/) (2018)
- [Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers](https://mlsanity.com/paper/1306.4447/) (2013)
- [Technical Report on the CleverHans v2.1.0 Adversarial Examples Library](https://mlsanity.com/paper/1610.00768/) (2016)
- [Adversarial Machine Learning -- Industry Perspectives](https://mlsanity.com/paper/2002.05646/) (2020)
- [Sponge Examples: Energy-Latency Attacks on Neural Networks](https://mlsanity.com/paper/2006.03463/) (2020)
- [Is feature selection secure against training data poisoning?](https://mlsanity.com/paper/1804.07933/) (2018)
- [Exploring Connections Between Active Learning and Model Extraction](https://mlsanity.com/paper/1811.02054/) (2018)
- [Poisoning Attacks on Cyber Attack Detectors for Industrial Control Systems](https://mlsanity.com/paper/2012.15740/) (2020)
- [Query-Efficient Black-box Adversarial Examples (superceded)](https://mlsanity.com/paper/1712.07113/) (2017)
- [On managing vulnerabilities in AI/ML systems](https://mlsanity.com/paper/2101.10865/) (2021)
- [Detection of Malicious and Low Throughput Data Exfiltration Over the DNS Protocol](https://mlsanity.com/paper/1709.08395/) (2017)

## Built on by (most cited first, 6 of 6)

- [Targeted Adversarial Attacks on Wind Power Forecasts](https://mlsanity.com/paper/2303.16633/) (2023)
- [Adversarial Machine Learning Threat Analysis and Remediation in Open Radio Access Network (O-RAN)](https://mlsanity.com/paper/2201.06093/) (2022)
- [Rogue Cell: Adversarial Attack and Defense in Untrusted O-RAN Setup Exploiting the Traffic Steering xApp](https://mlsanity.com/paper/2505.01816/) (2025)
- [An Investigation of Visual Foundation Models Robustness](https://mlsanity.com/paper/2508.16225/) (2025)
- [Learning to Collaborate: An Orchestrated-Decentralized Framework for Peer-to-Peer LLM Federation](https://mlsanity.com/paper/2601.17133/) (2026)
- [A Deterministic and Auditable AI Security Risk Assessment Framework with ATLAS Aligned Executable Rules and Formal Verification](https://mlsanity.com/paper/2610.01436/) (2026)

## Abstract (arXiv)

Although cyberattacks on machine learning (ML) production systems can be harmful, today, security practitioners are ill equipped, lacking methodologies and tactical tools that would allow them to analyze the security risks of their ML-based systems. In this paper, we performed a comprehensive threat analysis of ML production systems. In this analysis, we follow the ontology presented by NIST for evaluating enterprise network security risk and apply it to ML-based production systems. Specifically, we (1) enumerate the assets of a typical ML production system, (2) describe the threat model (i.e., potential adversaries, their capabilities, and their main goal), (3) identify the various threats to ML systems, and (4) review a large number of attacks, demonstrated in previous studies, which can realize these threats. In addition, to quantify the risk of adversarial machine learning (AML) threat, we introduce a novel scoring system, which assign a severity score to different AML attacks. The proposed scoring system utilizes the analytic hierarchy process (AHP) for ranking, with the assistance of security experts, various attributes of the attacks. Finally, we developed an extension to the MulVAL attack graph generation and analysis framework to incorporate cyberattacks on ML production systems. Using the extension, security practitioners can apply attack graph analysis methods in environments that include ML components; thus, providing security practitioners with a methodological and practical tool for evaluating the impact and quantifying the risk of a cyberattack targeting an ML production system.