mlsanityData to 14 Oct 2026

Towards more Practical Threat Models in Artificial Intelligence Security

arXiv 2311.09994 ↗First posted 16 Nov 2023cs.CR cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

Recent works have identified a gap between research and practice in artificial intelligence security: threats studied in academia do not always reflect the practical use and security risks of AI. For example, while models are often studied in isolation, they form part of larger ML pipelines in practice. Recent works also brought forward that adversarial manipulations introduced by academic attacks are impractical. We take a first step towards describing the full extent of this disparity. To this end, we revisit the threat models of the six most studied attacks in AI security research and match them to AI usage in practice via a survey with 271 industrial practitioners. On the one hand, we find that all existing threat models are indeed applicable. On the other hand, there are significant mismatches: research is often too generous with the attacker, assuming access to information not frequently available in real-world settings. Our paper is thus a call for action to study more practical threat models in artificial intelligence security.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 45 of its 80 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

45 of 80 references.

References · 45

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  2. 2013Intriguing properties of neural networkscited by 5,155
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  6. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  7. 2016Practical Black-Box Attacks against Machine Learningcited by 903
  8. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
  9. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  10. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  11. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  12. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
33 more
  1. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 414
  2. 2020Label-Only Membership Inference Attackscited by 388
  3. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  4. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 163
  5. 2018Scaling provable adversarial defensescited by 143
  6. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 138
  7. 2013Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifierscited by 130
  8. 2020Sponge Examples: Energy-Latency Attacks on Neural Networkscited by 111
  9. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 94
  10. 2018Is feature selection secure against training data poisoning?cited by 93
  11. 2017Vulnerability of Deep Reinforcement Learning to Policy Induction Attackscited by 91
  12. 2022I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defencescited by 75
  13. 2019Bypassing Backdoor Detection Algorithms in Deep Learningcited by 74
  14. 2021Manipulating SGD with Data Ordering Attackscited by 74
  15. 2018Adversarial Examples that Fool both Computer Vision and Time-Limited Humanscited by 73
  16. 2021Handcrafted Backdoors in Deep Neural Networkscited by 63
  17. 2020A Panda? No, It's a Sloth: Slowdown Attacks on Adaptive Multi-Exit Neural Network Inferencecited by 61
  18. 2018Motivating the Rules of the Game for Adversarial Example Researchcited by 57
  19. 2021Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Processcited by 56
  20. 2021A Backdoor Attack against 3D Point Cloud Classifierscited by 53
  21. 2021Poisoning the Unlabeled Dataset of Semi-Supervised Learningcited by 44
  22. 2021Survey: Leakage and Privacy at Inference Timecited by 44
  23. 2021Optical Adversarial Attackcited by 43
  24. 2022Physical Backdoor Attacks to Lane Detection Systems in Autonomous Drivingcited by 31
  25. 2022Energy-Latency Attacks via Sponge Poisoningcited by 29
  26. 2020Stealing Deep Reinforcement Learning Models for Fun and Profitcited by 24
  27. 2022Dangerous Cloaking: Natural Trigger based Backdoor Attacks on Object Detectors in the Physical Worldcited by 22
  28. 2022Machine Learning Security in Industry: A Quantitative Surveycited by 21
  29. 2021Back in Black: A Comparative Evaluation of Recent State-Of-The-Art Black-Box Attackscited by 16
  30. 2022Membership-Doctor: Comprehensive Assessment of Membership Inference Against Machine Learning Modelscited by 16
  31. 2021Adapting Software Architectures to Machine Learning Challengescited by 11
  32. 2020Security and Machine Learning in the Real Worldcited by 9
  33. 2021Membership Inference Attacks Against Temporally Correlated Data in Deep Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 6.

17 papers.

Sort:

By area:

  1. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  2. 2025The Hidden Dangers of Browsing AI Agentscited by 11
  3. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  4. 2025CRISP: Persistent Concept Unlearning via Sparse Autoencoderscited by 11
  5. 2025FlowPure: Continuous Normalizing Flows for Adversarial Purificationcited by 2
  6. 2024A Novel Perturb-ability Score to Mitigate Evasion Adversarial Attacks on Flow-Based ML-NIDScited by 1
  7. 2024Taking off the Rose-Tinted Glasses: A Critical Look at Adversarial ML Through the Lens of Evasion Attackscited by 1
  8. 2025Dimensional Characterization and Pathway Modeling for Catastrophic AI Riskscited by 1
  9. 2025Backdoor Mitigation via Invertible Pruning Maskscited by 1
  10. 2026BadDet+: Robust Backdoor Attacks for Object Detectioncited by 1
  11. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1
  12. 2025Expanding ML-Documentation Standards For Better Securitycited by 0
5 more
  1. 2025Surveying the Operational Cybersecurity and Supply Chain Threat Landscape when Developing and Deploying AI Systemscited by 0
  2. 2025I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attackscited by 0
  3. 2026A No-Defense Defense Against Gradient-Based Adversarial Attacks on ML-NIDS: Is Less More?cited by 0
  4. 2026DIPBox: A Multi-scale Testing Framework for Tracking Dataset Regenerationcited by 0
  5. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0