# Jailbreaking Generative AI: Empowering Novices to Conduct Phishing Attacks

arXiv: https://arxiv.org/abs/2503.01395 · HTML: https://mlsanity.com/paper/2503.01395/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 3 Mar 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 4 of 4)

- [MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbots](https://mlsanity.com/paper/2307.08715/) (2023)
- [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://mlsanity.com/paper/2307.00691/) (2023)
- [Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT](https://mlsanity.com/paper/2309.10463/) (2023)
- [AbuseGPT: Abuse of Generative AI ChatBots to Create Smishing Campaigns](https://mlsanity.com/paper/2402.09728/) (2024)

## Built on by (most cited first, 1 of 1)

- [Jailbreaking Generative AI: Multivector Phishing Threats and Transformer based Defenses](https://mlsanity.com/paper/2507.12185/) (2025)

## Abstract (arXiv)

The rapid advancements in generative AI models, such as ChatGPT, have introduced both significant benefits and new risks within the cybersecurity landscape. This paper investigates the potential misuse of the latest AI model, ChatGPT-4o Mini, in facilitating social engineering attacks, with a particular focus on phishing, one of the most pressing cybersecurity threats today. While existing literature primarily addresses the technical aspects, such as jailbreaking techniques, none have fully explored the free and straightforward execution of a comprehensive phishing campaign by novice users using ChatGPT-4o Mini. In this study, we examine the vulnerabilities of AI-driven chatbot services in 2025, specifically how methods like jailbreaking and reverse psychology can bypass ethical safeguards, allowing ChatGPT to generate phishing content, suggest hacking tools, and assist in carrying out phishing attacks. Our findings underscore the alarming ease with which even inexperienced users can execute sophisticated phishing campaigns, emphasizing the urgent need for stronger cybersecurity measures and heightened user awareness in the age of AI.