# PhishAgent: A Robust Multimodal Agent for Phishing Webpage Detection

arXiv: https://arxiv.org/abs/2408.10738 · HTML: https://mlsanity.com/paper/2408.10738/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Aug 2024.

## Built on by (most cited first, 19 of 19)

- [VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents](https://mlsanity.com/paper/2506.02456/) (2025)
- [Towards Trustworthy GUI Agents: A Survey](https://mlsanity.com/paper/2503.23434/) (2025)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [Frontier AI's Impact on the Cybersecurity Landscape](https://mlsanity.com/paper/2504.05408/) (2025)
- [TopicAttack: An Indirect Prompt Injection Attack via Topic Transition](https://mlsanity.com/paper/2507.13686/) (2025)
- [PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMs](https://mlsanity.com/paper/2508.17188/) (2025)
- [WARD: Adversarially Robust Defense of Web Agents Against Prompt Injections](https://mlsanity.com/paper/2605.15030/) (2026)
- [Backdoor-Powered Prompt Injection Attacks Nullify Defense Methods](https://mlsanity.com/paper/2510.03705/) (2025)
- [WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agents](https://mlsanity.com/paper/2604.12284/) (2026)
- [PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark](https://mlsanity.com/paper/2507.10854/) (2025)
- [PreScam: A Benchmark for Predicting Scam Progression from Early Conversations](https://mlsanity.com/paper/2605.12243/) (2026)
- [WebGuard++:Interpretable Malicious URL Detection via Bidirectional Fusion of HTML Subgraphs and Multi-Scale Convolutional BERT](https://mlsanity.com/paper/2506.19356/) (2025)
- [MemoPhishAgent: Memory-Augmented Multi-Modal LLM Agent for Phishing URL Detection](https://mlsanity.com/paper/2602.21394/) (2026)
- [Phishing the Phishers with SpecularNet: Hierarchical Graph Autoencoding for Reference-Free Web Phishing Detection](https://mlsanity.com/paper/2603.01874/) (2026)
- [AliMark: Enhancing Robustness of Sentence-Level Watermarking Against Text Paraphrasing](https://mlsanity.com/paper/2605.29434/) (2026)
- [PhishLumos: An Adaptive Multi-Agent System for Proactive Phishing Campaign Mitigation](https://mlsanity.com/paper/2509.21772/) (2025)
- [Role-Aware Multi-modal federated learning system for detecting phishing webpages](https://mlsanity.com/paper/2509.22369/) (2025)
- [SCRIPTMIND: Crime Script Inference and Cognitive Evaluation for LLM-based Social Engineering Scam Detection System](https://mlsanity.com/paper/2601.13581/) (2026)
- [Clouding the Mirror: Stealthy Prompt Injection Attacks Targeting LLM-based Phishing Detection](https://mlsanity.com/paper/2602.05484/) (2026)

## Abstract (arXiv)

Phishing attacks are a major threat to online security, exploiting user vulnerabilities to steal sensitive information. Various methods have been developed to counteract phishing, each with varying levels of accuracy, but they also face notable limitations. In this study, we introduce PhishAgent, a multimodal agent that combines a wide range of tools, integrating both online and offline knowledge bases with Multimodal Large Language Models (MLLMs). This combination leads to broader brand coverage, which enhances brand recognition and recall. Furthermore, we propose a multimodal information retrieval framework designed to extract the relevant top k items from offline knowledge bases, using available information from a webpage, including logos and HTML. Our empirical results, based on three real-world datasets, demonstrate that the proposed framework significantly enhances detection accuracy and reduces both false positives and false negatives, while maintaining model efficiency. Additionally, PhishAgent shows strong resilience against various types of adversarial attacks.