# Assessing Spear-Phishing Website Generation in Large Language Model Coding Agents

arXiv: https://arxiv.org/abs/2602.13363 · HTML: https://mlsanity.com/paper/2602.13363/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 13 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 17 of 17)

- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [LLM Agents can Autonomously Hack Websites](https://mlsanity.com/paper/2402.06664/) (2024)
- [WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment](https://mlsanity.com/paper/2402.12275/) (2024)
- [Spear Phishing With Large Language Models](https://mlsanity.com/paper/2305.06972/) (2023)
- [AutoAgent: A Fully-Automated and Zero-Code Framework for LLM Agents](https://mlsanity.com/paper/2502.05957/) (2025)
- [Large Language Models Pass the Turing Test](https://mlsanity.com/paper/2503.23674/) (2025)
- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [LLMs in Web Development: Evaluating LLM-Generated PHP Code Unveiling Vulnerabilities and Limitations](https://mlsanity.com/paper/2404.14459/) (2024)
- [Next-Generation Phishing: How LLM Agents Empower Cyber Attackers](https://mlsanity.com/paper/2411.13874/) (2024)
- [Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detection](https://mlsanity.com/paper/2503.15552/) (2025)
- [Copilot for Xcode: Exploring AI-Assisted Programming by Prompting Cloud-based Large Language Models](https://mlsanity.com/paper/2307.14349/) (2023)
- [Breaching the Human Firewall: Social engineering in Phishing and Spear-Phishing Emails](https://mlsanity.com/paper/1606.00887/) (2016)
- [Prompted Contextual Vectors for Spear-Phishing Detection](https://mlsanity.com/paper/2402.08309/) (2024)
- [The Impact of Emerging Phishing Threats: Assessing Quishing and LLM-generated Phishing Emails against Organizations](https://mlsanity.com/paper/2505.12104/) (2025)
- [Training Users Against Human and GPT-4 Generated Social Engineering Attacks](https://mlsanity.com/paper/2502.01764/) (2025)
- [PiMRef: Deducing Ever-evolving Spear-phishing Emails with Knowledge Base Invariants](https://mlsanity.com/paper/2507.15393/) (2025)

## Abstract (arXiv)

Large Language Models are expanding beyond being a tool humans use and into independent agents that can observe an environment, reason about solutions to problems, make changes that impact those environments, and understand how their actions impacted their environment. One of the most common applications of these LLM Agents is in computer programming, where agents can successfully work alongside humans to generate code while controlling programming environments or networking systems. However, with the increasing ability and complexity of these agents comes dangers about the potential for their misuse. A concerning application of LLM agents is in the domain cybersecurity, where they have the potential to greatly expand the threat imposed by attacks such as social engineering. This is due to the fact that LLM Agents can work autonomously and perform many tasks that would normally require time and effort from skilled human programmers. While this threat is concerning, little attention has been given to assessments of the capabilities of LLM coding agents in generating code for social engineering attacks. In this work we compare different LLMs in their ability and willingness to produce potentially dangerous code bases that could be misused by cyberattackers. The result is a dataset of 200 website code bases and logs from 40 different LLM coding agents. Analysis of models shows which metrics of LLMs are more and less correlated with performance in generating spear-phishing sites. Our analysis and the dataset we present will be of interest to researchers and practitioners concerned in defending against the potential misuse of LLMs in spear-phishing.