# The Phish, The Spam, and The Valid: Generating Feature-Rich Emails for Benchmarking LLMs

arXiv: https://arxiv.org/abs/2511.21448 · HTML: https://mlsanity.com/paper/2511.21448/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 8 of 8)

- [Next-Generation Phishing: How LLM Agents Empower Cyber Attackers](https://mlsanity.com/paper/2411.13874/) (2024)
- [Enhancing Phishing Email Identification with Large Language Models](https://mlsanity.com/paper/2502.04759/) (2025)
- [Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence](https://mlsanity.com/paper/2410.15490/) (2024)
- [E-PhishGen: Unlocking Novel Research in Phishing Email Detection](https://mlsanity.com/paper/2509.01791/) (2025)
- [SecureNet: A Comparative Study of DeBERTa and Large Language Models for Phishing Detection](https://mlsanity.com/paper/2406.06663/) (2024)
- [Phishing Codebook: A Structured Framework for the Characterization of Phishing Emails](https://mlsanity.com/paper/2408.08967/) (2024)
- [LLM-Powered Intent-Based Categorization of Phishing Emails](https://mlsanity.com/paper/2506.14337/) (2025)
- [Sustaining Cyber Awareness: The Long-Term Impact of Continuous Phishing Training and Emotional Triggers](https://mlsanity.com/paper/2510.27298/) (2025)

## Built on by (most cited first, 4 of 4)

- [Are Concept Bottleneck Models Effective as Decision-Support Systems?](https://mlsanity.com/paper/2608.25581/) (2026)
- [A Hybrid, Multi-Layered Pipeline for Phishing and Threat Classification: Independently Validated URL and NLP Engines with a Calibrated Multi-Channel Fusion Stage](https://mlsanity.com/paper/2606.21690/) (2026)
- [Evaluating Agentic Learning Harness Capabilities Without Labels via the Scaling Hypothesis](https://mlsanity.com/paper/2608.13608/) (2026)
- [AURA: Adaptive Uncertainty-Routed Analysis for Email Threat Detection](https://mlsanity.com/paper/2609.19873/) (2026)

## Abstract (arXiv)

In this paper, we introduce a metadata-enriched generation framework (PhishFuzzer) that seeds real emails into Large Language Models (LLMs) to produce 23,100 diverse, structurally consistent email variants across controlled entity and length dimensions. Unlike prior corpora, our dataset features strict three-class labels (Phishing, Spam, Valid), provides full URL and attachment metadata, and annotates each email with attacker intent. Using this dataset, we benchmark two state-of-the-art LLMs (Qwen-2.5-72B and Gemini-3.1-Pro) under both Basic (body, subject) and Full (+URL, sender, attachment) settings. By applying formal confidence metrics (Task Success Rate and Confidence Index), we analyze model reliability, robustness against linguistic fuzzing, and the impact of structural metadata on detection accuracy. Our fully open-source framework and dataset provide a rigorous foundation for evaluating next-generation email security systems. To support open science, we make the PhishFuzzer Dataset, the generation scripts and prompts available on GitHub: https://github.com/DataPhish/PhishFuzzer