mlsanityData to 7 Oct 2026

Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data

arXiv 2406.13843 ↗First posted 19 Jun 2024cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Generative, multimodal artificial intelligence (GenAI) offers transformative potential across industries, but its misuse poses significant risks. Prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes. However, we still lack a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, informed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023 and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

32 of 51 references.

References · 32

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  10. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  12. 2023Can AI-Generated Text be Reliably Detected?cited by 372
20 more
  1. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  2. 2023Model evaluation for extreme riskscited by 167
  3. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  4. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  5. 2024Stealing Part of a Production Language Modelcited by 152
  6. 2023In-Context Learning Unlocked for Diffusion Modelscited by 103
  7. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  8. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  9. 2023BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languagescited by 60
  10. 2022Structured access: an emerging paradigm for safe AI deploymentcited by 40
  11. 2023GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Modelscited by 40
  12. 2024AMMeBa: A Large-Scale Survey and Dataset of Media-Based Misinformation In-The-Wildcited by 26
  13. 2024Malla: Demystifying Real-world Large Language Model Integrated Malicious Servicescited by 24
  14. 2019Adapting Language Models for Non-Parallel Author-Stylized Rewritingcited by 23
  15. 2024As Good As A Coin Toss: Human detection of AI-generated images, videos, audio, and audiovisual stimulicited by 22
  16. 2023Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted?cited by 15
  17. 2024How Spammers and Scammers Leverage AI-Generated Images on Facebook for Audience Growthcited by 11
  18. 2024Generative AI is already widespread in the public sectorcited by 10
  19. 2018Unpacking the Social Media Bot: A Typology to Guide Research and Policycited by 7
  20. 2021The Deepfake Detection Dilemma: A Multistakeholder Exploration of Adversarial Dynamics in Synthetic Mediacited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 12; 2026: 2.

20 papers.

Sort:

By area:

  1. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  2. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  3. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  4. 2024Acceptable Use Policies for Foundation Modelscited by 12
  5. 2024Analyzing the AI Nudification Application Ecosystemcited by 9
  6. 2025OpenFake: An Open Dataset and Platform Toward Real-World Deepfake Detectioncited by 9
  7. 2025Model Immunization from a Condition Number Perspectivecited by 8
  8. 2025In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AIcited by 7
  9. 2024AI and the Future of Digital Public Squarescited by 6
  10. 2025Real-World Gaps in AI Governance Researchcited by 5
  11. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  12. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 3
8 more
  1. 2025ClusterMark: Towards Robust Watermarking for Autoregressive Image Generators with Visual Token Clusteringcited by 3
  2. 2025Training-free Source Attribution of AI-generated Images via Resynthesiscited by 3
  3. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  4. 2025SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generationcited by 2
  5. 2026Designing to Forget: Deep Semi-parametric Models for Unlearningcited by 1
  6. 2024Responsible AI in Open Ecosystems: Reconciling Innovation with Risk Assessment and Disclosurecited by 0
  7. 2025Datum-wise Transformer for Synthetic Tabular Data Detection in the Wildcited by 0
  8. 2026Understanding U.S. Users' Security and Privacy Transparency Needs for Consumer-Facing Generative AIcited by 0