mlsanityData to 7 Oct 2026

SoK: On the Offensive Potential of AI

arXiv 2412.18442 ↗First posted 24 Dec 2024cs.CR cs.AI cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

Our society increasingly benefits from Artificial Intelligence (AI). Unfortunately, more and more evidence shows that AI is also used for offensive purposes. Prior works have revealed various examples of use cases in which the deployment of AI can lead to violation of security and privacy objectives. No extant work, however, has been able to draw a holistic picture of the offensive potential of AI. In this SoK paper we seek to lay the ground for a systematic analysis of the heterogeneous capabilities of offensive AI. In particular we (i) account for AI risks to both humans and systems while (ii) consolidating and distilling knowledge from academic literature, expert opinions, industrial venues, as well as laypeople -- all of which being valuable sources of information on offensive AI. To enable alignment of such diverse sources of knowledge, we devise a common set of criteria reflecting essential technological factors related to offensive AI. With the help of such criteria, we systematically analyze: 95 research papers; 38 InfoSec briefings (from, e.g., BlackHat); the responses of a user study (N=549) entailing individuals with diverse backgrounds and expertise; and the opinion of 12 experts. Our contributions not only reveal concerning ways (some of which overlooked by prior work) in which AI can be offensively used today, but also represent a foothold to address this threat in the years to come.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 50 of its 249 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

50 of 249 references.

References · 50

  1. 2019Advances and Open Problems in Federated Learningcited by 3,021
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  6. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  7. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  8. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  9. 2022DE-FAKE: Detection and Attribution of Fake Images Generated by Text-to-Image Generation Modelscited by 167
  10. 2023Getting pwn'd by AI: Penetration Testing with Large Language Modelscited by 82
  11. 2017Generating Adversarial Malware Examples for Black-Box Attacks Based on GANcited by 74
  12. 2018Learning to Evade Static PE Machine Learning Malware Models via Reinforcement Learningcited by 60
38 more
  1. 2021CybORG: A Gym for the Development of Autonomous Cyber Agentscited by 58
  2. 2022Taxonomy of Attacks on Open-Source Software Supply Chainscited by 57
  3. 2022"Real Attackers Don't Compute Gradients": Bridging the Gap Between Adversarial ML Research and Practicecited by 55
  4. 2018IDSGAN: Generative Adversarial Networks for Attack Generation against Intrusion Detectioncited by 41
  5. 2021The Threat of Offensive AI to Organizationscited by 37
  6. 2016You are Who You Know and How You Behave: Attribute Inference Attacks via Users' Social Friends and Behaviorscited by 34
  7. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  8. 2017Automated Website Fingerprinting through Deep Learningcited by 21
  9. 2022The Role of Machine Learning in Cybersecuritycited by 21
  10. 2021Deep hierarchical reinforcement agents for automated penetration testingcited by 19
  11. 2023Devising and Detecting Phishing: Large Language Models vs. Smaller Human Modelscited by 17
  12. 2021Using Cyber Terrain in Reinforcement Learning for Penetration Testingcited by 16
  13. 2022SoK: The Impact of Unlabelled Data in Cyberthreat Detectioncited by 15
  14. 2016DeepDGA: Adversarially-Tuned Domain Generation and Detectioncited by 14
  15. 2023SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detectioncited by 13
  16. 2017Automated Crowdturfing Attacks and Defenses in Online Review Systemscited by 12
  17. 2015When Coding Style Survives Compilation: De-anonymizing Programmers from Executable Binariescited by 11
  18. 2020DeepSQLi: Deep Semantic Learning for Testing SQL Injectioncited by 11
  19. 2022Mimetic Models: Ethical Implications of AI that Acts Like Youcited by 11
  20. 2021PassFlow: Guessing Passwords with Generative Flowscited by 10
  21. 2021Crown Jewels Analysis using Reinforcement Learning with Attack Graphscited by 8
  22. 2022Discovering Exfiltration Paths Using Reinforcement Learning with Attack Graphscited by 8
  23. 2022Multi-SpacePhish: Extending the Evasion-space of Adversarial Attacks against Phishing Website Detectors using Machine Learningcited by 7
  24. 2021Simulating SQL Injection Vulnerability Exploitation Using Q-Learning Reinforcement Learning Agentscited by 6
  25. 2013Studying User Footprints in Different Online Social Networkscited by 5
  26. 2021Automating Privilege Escalation with Deep Reinforcement Learningcited by 5
  27. 2022Attribute Inference Attacks in Online Multiplayer Video Games: a Case Study on Dota2cited by 4
  28. 2020Automated Adversary Emulation for Cyber-Physical Systems via Reinforcement Learningcited by 3
  29. 2023"Do Users fall for Real Adversarial Phishing?" Investigating the Human response to Evasive Webpagescited by 3
  30. 2017Community Targeted Phishing: A Middle Ground Between Massive and Spear Phishing through Natural Language Generationcited by 2
  31. 2020AI-Powered GUI Attack and Its Defensive Methodscited by 2
  32. 2020Machine Learning for Offensive Security: Sandbox Classification Using Decision Trees and Artificial Neural Networkscited by 2
  33. 2020Reinforcement Learning Based Penetration Testing of a Microgrid Control Algorithmcited by 2
  34. 2020My tweets bring all the traits to the yard: Predicting personality and relational traits in Online Social Networkscited by 2
  35. 2021Catch Me If You GAN: Using Artificial Intelligence for Fake Log Generationcited by 2
  36. 2019AppMine: Behavioral Analytics for Web Application Vulnerability Detectioncited by 1
  37. 2020Efficient Quantification of Profile Matching Risk in Social Networkscited by 1
  38. 2021Leveraging AI to optimize website structure discovery during Penetration Testingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Frontier AI's Impact on the Cybersecurity Landscapecited by 21
  2. 2025Incalmo: An Autonomous LLM-assisted System for Red Teaming Multi-Host Networkscited by 17
  3. 2026Adversarial News and Lost Profits: Manipulating Headlines in LLM-Driven Algorithmic Tradingcited by 4
  4. 2025Graph of Effort: Quantifying Risk of AI Usage for Vulnerability Assessmentcited by 1
  5. 2026From Beats to Breaches:How Offensive AI Infers Sensitive User Information from Playlistscited by 0