mlsanityData to 7 Oct 2026

The Threat of Offensive AI to Organizations

arXiv 2106.15764 ↗First posted 30 Jun 2021cs.AI cs.CR cs.CYcited by 37

Not on the map. Why?

Abstract (arXiv)

AI has provided us with the ability to automate tasks, extract information from vast amounts of data, and synthesize media that is nearly indistinguishable from the real thing. However, positive tools can also be used for negative purposes. In particular, cyber adversaries can use AI (such as machine learning) to enhance their attacks and expand their campaigns. Although offensive AI has been discussed in the past, there is a need to analyze and understand the threat in the context of organizations. For example, how does an AI-capable adversary impact the cyber kill chain? Does AI benefit the attacker more than the defender? What are the most significant AI threats facing organizations today and what will be their impact on the future? In this survey, we explore the threat of offensive AI on organizations. First, we present the background and discuss how AI changes the adversary's methods, strategies, goals, and overall attack model. Then, through a literature review, we identify 33 offensive AI capabilities which adversaries can use to enhance their attacks. Finally, through a user study spanning industry and academia, we rank the AI threats and provide insights on the adversaries.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 86 of its 275 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

86 of 275 references.

References · 86

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  7. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  9. 2018Link Prediction Based on Graph Neural Networkscited by 829
  10. 2020First Order Motion Model for Image Animationcited by 698
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  12. 2020Recipes for building an open-domain chatbotcited by 593
38 more
  1. 2019Graph Transformer Networkscited by 520
  2. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  3. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  4. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  5. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  6. 2019FSGAN: Subject Agnostic Face Swapping and Reenactmentcited by 319
  7. 2018Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesiscited by 280
  8. 2018Adversarial Attacks and Defences: A Surveycited by 259
  9. 2020The Creation and Detection of Deepfakes: A Surveycited by 246
  10. 2018Adversarial Robustness Toolbox v1.0.0cited by 214
  11. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  12. 2014Graph-based Anomaly Detection and Description: A Surveycited by 182
  13. 2020Blind Backdoors in Deep Learning Modelscited by 174
  14. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  15. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
  16. 2017Style Transfer in Text: Exploration and Evaluationcited by 145
  17. 2020Entangled Watermarks as a Defense against Model Extractioncited by 144
  18. 2020Deep Learning based Vulnerability Detection: Are We There Yet?cited by 142
  19. 2016Technical Report on the CleverHans v2.1.0 Adversarial Examples Librarycited by 123
  20. 2018Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detectioncited by 121
  21. 2018Stealing Hyperparameters in Machine Learningcited by 115
  22. 2020Sponge Examples: Energy-Latency Attacks on Neural Networkscited by 108
  23. 2019A Survey of Deep Learning-based Object Detectioncited by 106
  24. 2019A Deep Look into Neural Ranking Models for Information Retrievalcited by 79
  25. 2018Unsupervised Text Style Transfer using Language Models as Discriminatorscited by 70
  26. 2017Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detectioncited by 65
  27. 2020Bullseye Polytope: A Scalable Clean-Label Poisoning Attack with Improved Transferabilitycited by 65
  28. 2019The Limitations of Adversarial Training and the Blind-Spot Attackcited by 63
  29. 2018Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learningcited by 60
  30. 2018Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executablescited by 60
  31. 2020Functionality-preserving Black-box Optimization of Adversarial Windows Malwarecited by 58
  32. 2018A Hybrid Model for Identity Obfuscation by Face Replacementcited by 43
  33. 2020Backdoor Attacks against Transfer Learning with Pre-trained Deep Learning Modelscited by 43
  34. 2018Exploring Adversarial Examples in Malware Detectioncited by 41
  35. 2019Almost Unsupervised Text to Speech and Automatic Speech Recognitioncited by 41
  36. 2020VulDeeLocator: A Deep Learning-based Fine-grained Vulnerability Detectorcited by 37
  37. 2019Autonomous Penetration Testing using Reinforcement Learningcited by 33
  38. 2017Survey of Machine Learning Techniques for Malware Analysiscited by 31

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 2; 2023: 15; 2024: 7; 2025: 9; 2026: 1.

37 papers.

Sort:

By area:

  1. 2023A Watermark for Large Language Modelscited by 667
  2. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  3. 2023An Overview of Catastrophic AI Riskscited by 214
  4. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  5. 2023On the Exploitability of Instruction Tuningcited by 110
  6. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 102
  7. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  8. 2024Raidar: geneRative AI Detection viA Rewritingcited by 51
  9. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  10. 2024GumbelSoft: Diversified Language Model Watermarking via the GumbelMax-trickcited by 31
  11. 2022Machine Learning Security in Industry: A Quantitative Surveycited by 21
  12. 2025CoheMark: A Novel Sentence-Level Watermark for Enhanced Text Qualitycited by 17
25 more
  1. 2021EvilModel 2.0: Bringing Neural Network Models into Malware Attackscited by 13
  2. 2022Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generatorscited by 10
  3. 2023Towards an Understanding and Explanation for Mixed-Initiative Artificial Scientific Text Detectioncited by 8
  4. 2023Impacts and Risk of Generative AI Technology on Cyber Defensecited by 8
  5. 2023Adversarial Machine Learning for Social Good: Reframing the Adversary as an Allycited by 5
  6. 2024MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviewscited by 5
  7. 2024SoK: On the Offensive Potential of AIcited by 5
  8. 2025Hiding in the AI Traffic: Abusing MCP for LLM-Powered Agentic Red Teamingcited by 4
  9. 2026PASA: A Principled Embedding-Space Watermarking Approach for LLM-Generated Text under Semantic-Invariant Attackscited by 4
  10. 2023Enhancing Robustness of AI Offensive Code Generators via Data Augmentationcited by 3
  11. 2023You Don't Need Robust Machine Learning to Manage Adversarial Attack Riskscited by 3
  12. 2023A Red Teaming Framework for Securing AI in Maritime Autonomous Systemscited by 3
  13. 2024Enhancing AI-based Generation of Software Exploits with Contextual Informationcited by 3
  14. 2024GigaCheck: Detecting LLM-generated Content via Object-Centric Span Localizationcited by 3
  15. 2025Addressing Tokenization Inconsistency in Steganography and Watermarking Based on Large Language Modelscited by 3
  16. 2024The Best Defense is a Good Offense: Countering LLM-Powered Cyberattackscited by 2
  17. 2025IPAD: Inverse Prompt for AI Detection - A Robust and Interpretable LLM-Generated Text Detectorcited by 2
  18. 2025"We need to avail ourselves of GenAI to enhance knowledge distribution": Empowering Older Adults through GenAI Literacycited by 2
  19. 2025Graph of Effort: Quantifying Risk of AI Usage for Vulnerability Assessmentcited by 1
  20. 2025Unintentional Consequences: Generative AI Use for Cybercrimecited by 1
  21. 2025Black-Box Detection of LLM-Generated Text Using Generalized Jensen-Shannon Divergencecited by 1
  22. 2021Color Teams for Machine Learning Developmentcited by 0
  23. 2021Phantom: A High-Performance Computational Core for Sparse Convolutional Neural Networkscited by 0
  24. 2023Threats, Vulnerabilities, and Controls of Machine Learning Based Systems: A Survey and Taxonomycited by 0
  25. 2025Temperature Matters: Enhancing Watermark Robustness Against Paraphrasing Attackscited by 0