# Evaluating the Efficacy of Large Language Models in Identifying Phishing Attempts

arXiv: https://arxiv.org/abs/2404.15485 · HTML: https://mlsanity.com/paper/2404.15485/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 23 Apr 2024.

## Builds on (most cited first, 16 of 16)

- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Pretrained Language Models for Text Generation: A Survey](https://mlsanity.com/paper/2105.10311/) (2021)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extraction](https://mlsanity.com/paper/1906.08646/) (2019)
- [A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutions](https://mlsanity.com/paper/2305.18339/) (2023)
- [BioinspiredLLM: Conversational Large Language Model for the Mechanics of Biological and Bio-inspired Materials](https://mlsanity.com/paper/2309.08788/) (2023)
- [SPEC5G: A Dataset for 5G Cellular Network Protocol Analysis](https://mlsanity.com/paper/2301.09201/) (2023)
- [Balancing Specialized and General Skills in LLMs: The Impact of Modern Tuning and Data Strategy](https://mlsanity.com/paper/2310.04945/) (2023)
- [Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition](https://mlsanity.com/paper/2108.07789/) (2021)
- [ChatGPT in the Age of Generative AI and Large Language Models: A Concise Survey](https://mlsanity.com/paper/2307.04251/) (2023)
- [Hallucination-minimized Data-to-answer Framework for Financial Decision-makers](https://mlsanity.com/paper/2311.07592/) (2023)
- [Transfer Fine-Tuning: A BERT Case Study](https://mlsanity.com/paper/1909.00931/) (2019)
- [The MalSource Dataset: Quantifying Complexity and Code Reuse in Malware Development](https://mlsanity.com/paper/1811.06888/) (2018)
- [OntoChatGPT Information System: Ontology-Driven Structured Prompts for ChatGPT Meta-Learning](https://mlsanity.com/paper/2307.05082/) (2023)

## Built on by (most cited first, 2 of 2)

- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [Trustworthiness Calibration Framework for Phishing Email Detection Using Large Language Models](https://mlsanity.com/paper/2511.04728/) (2025)

## Abstract (arXiv)

Phishing, a prevalent cybercrime tactic for decades, remains a significant threat in today's digital world. By leveraging clever social engineering elements and modern technology, cybercrime targets many individuals, businesses, and organizations to exploit trust and security. These cyber-attackers are often disguised in many trustworthy forms to appear as legitimate sources. By cleverly using psychological elements like urgency, fear, social proof, and other manipulative strategies, phishers can lure individuals into revealing sensitive and personalized information. Building on this pervasive issue within modern technology, this paper aims to analyze the effectiveness of 15 Large Language Models (LLMs) in detecting phishing attempts, specifically focusing on a randomized set of "419 Scam" emails. The objective is to determine which LLMs can accurately detect phishing emails by analyzing a text file containing email metadata based on predefined criteria. The experiment concluded that the following models, ChatGPT 3.5, GPT-3.5-Turbo-Instruct, and ChatGPT, were the most effective in detecting phishing emails.