mlsanityData to 14 Oct 2026

Spear Phishing With Large Language Models

arXiv 2305.06972 ↗First posted 11 May 2023cs.CY cs.AI cs.CRcited by 52

Not on the map. Why?

Abstract (arXiv)

Recent progress in artificial intelligence (AI), particularly in the domain of large language models (LLMs), has resulted in powerful and versatile dual-use systems. This intelligence can be put towards a wide variety of beneficial tasks, yet it can also be used to cause harm. This study explores one such harm by examining how LLMs can be used for spear phishing, a form of cybercrime that involves manipulating targets into divulging sensitive information. I first explore LLMs' ability to assist with the reconnaissance and message generation stages of a spear phishing attack, where I find that LLMs are capable of assisting with the email generation phase of a spear phishing attack. To explore how LLMs could potentially be harnessed to scale spear phishing campaigns, I then create unique spear phishing messages for over 600 British Members of Parliament using OpenAI's GPT-3.5 and GPT-4 models. My findings provide some evidence that these messages are not only realistic but also cost-effective, with each email costing only a fraction of a cent to generate. Next, I demonstrate how basic prompt engineering can circumvent safeguards installed in LLMs, highlighting the need for further research into robust interventions that can help prevent models from being misused. To further address these evolving risks, I explore two potential solutions: structured access schemes, such as application programming interfaces, and LLM-based defensive systems.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 9 of its 38 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

9 of 38 references.

References · 9

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  6. 2022Compute Trends Across Three Eras of Machine Learningcited by 163
  7. 2022Structured access: an emerging paradigm for safe AI deploymentcited by 41
  8. 2023Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted?cited by 16
  9. 2018Generative Models for Spear Phishing Posts on Social Mediacited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 14; 2024: 15; 2025: 11; 2026: 12.

52 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  3. 2023Watermarks in the Sand: Impossibility of Strong Watermarking for Generative Modelscited by 87
  4. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  5. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 65
  6. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  7. 2024Large Language Model Safety: A Holistic Surveycited by 59
  8. 2024CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Modelscited by 58
  9. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?cited by 55
  10. 2023Hoodwinked: Deception and Cooperation in a Text-Based Game for Language Modelscited by 47
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  12. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 37
38 more
  1. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 35
  2. 2023AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiologycited by 34
  3. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  4. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 27
  5. 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
  6. 2023Negotiating with LLMS: Prompt Hacks, Skill Gaps, and Reasoning Deficitscited by 16
  7. 2024Next-Generation Phishing: How LLM Agents Empower Cyber Attackerscited by 14
  8. 2024Lateral Phishing With Large Language Models: A Large Organization Comparative Studycited by 12
  9. 2023Transformation vs Tradition: Artificial General Intelligence (AGI) for Arts and Humanitiescited by 9
  10. 2024On TinyML and Cybersecurity: Electric Vehicle Charging Infrastructure Use Casecited by 9
  11. 2023International Governance of Civilian AI: A Jurisdictional Certification Approachcited by 8
  12. 2023Warfare:Breaking the Watermark Protection of AI-Generated Contentcited by 8
  13. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intentcited by 8
  14. 2026PASA: A Principled Embedding-Space Watermarking Approach for LLM-Generated Text under Semantic-Invariant Attackscited by 7
  15. 2025LLM Safety Alignment is Divergence Estimation in Disguisecited by 6
  16. 2024How will advanced AI systems impact democracy?cited by 5
  17. 2025Every Step Counts: Decoding Trajectories as Authorship Fingerprints of dLLMscited by 4
  18. 2026An Independent Safety Evaluation of Kimi K2.5cited by 3
  19. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  20. 2025Military AI Cyber Agents (MAICAs) Constitute a Global Threat to Critical Infrastructurecited by 2
  21. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuationcited by 2
  22. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  23. 2024Countering Autonomous Cyber Threatscited by 1
  24. 2025Cyri: A Conversational AI-based Assistant for Supporting the Human User in Detecting and Responding to Phishing Attackscited by 1
  25. 2025AI threats to national security can be countered through an incident regimecited by 1
  26. 2025Evaluating Large Language Models for Phishing Detection, Self-Consistency, Faithfulness, and Explainabilitycited by 1
  27. 2025CourtGuard: A Local, Multiagent Prompt Injection Classifiercited by 1
  28. 2026Multilingual jailbreaking of LLMs using low-resource languagescited by 1
  29. 2024Cryptocurrency Frauds for Dummies: How ChatGPT introduces us to fraud?cited by 0
  30. 2024A Lean Transformer Model for Dynamic Malware Analysis and Detectioncited by 0
  31. 2025Red Teaming with Artificial Intelligence-Driven Cyberattacks: A Scoping Reviewcited by 0
  32. 2025MultiPhishGuard: An Explainable and Adaptive Multi-Agent LLM System for Phishing Email Detectioncited by 0
  33. 2025Every Character Counts: From Vulnerability to Defense in Phishing Detectioncited by 0
  34. 2026Assessing Spear-Phishing Website Generation in Large Language Model Coding Agentscited by 0
  35. 2026Beyond A Fixed Seal: Adaptive Stealing Watermark in Large Language Modelscited by 0
  36. 2026The End of Trust: How Agentic AI Breaks Security Assumptionscited by 0
  37. 2026From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AIcited by 0
  38. 2026Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safetycited by 0

50 most cited of 52 All 52 (JSON)