# Multimodal Large Language Models for Phishing Webpage Detection and Identification

arXiv: https://arxiv.org/abs/2408.05941 · HTML: https://mlsanity.com/paper/2408.05941/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [AutoCodeRover: Autonomous Program Improvement](https://mlsanity.com/paper/2404.05427/) (2024)
- [Generative Adversarial Perturbations](https://mlsanity.com/paper/1712.02328/) (2017)
- [Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs](https://mlsanity.com/paper/2307.10490/) (2023)
- [Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation](https://mlsanity.com/paper/1806.01156/) (2018)
- [WIPI: A New Web Threat for LLM-Driven Web Agents](https://mlsanity.com/paper/2402.16965/) (2024)
- [You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content](https://mlsanity.com/paper/2308.05596/) (2023)
- [KnowPhish: Large Language Models Meet Multimodal Knowledge Graphs for Enhancing Reference-Based Phishing Detection](https://mlsanity.com/paper/2403.02253/) (2024)
- [ChatSpamDetector: Leveraging Large Language Models for Effective Phishing Email Detection](https://mlsanity.com/paper/2402.18093/) (2024)
- [LLMs for Cyber Security: New Opportunities](https://mlsanity.com/paper/2404.11338/) (2024)
- [Multi-SpacePhish: Extending the Evasion-space of Adversarial Attacks against Phishing Website Detectors using Machine Learning](https://mlsanity.com/paper/2210.13660/) (2022)
- [From ML to LLM: Evaluating the Robustness of Phishing Webpage Detection Models against Adversarial Attacks](https://mlsanity.com/paper/2407.20361/) (2024)
- [Evaluating the Effectiveness and Robustness of Visual Similarity-based Phishing Detection Models](https://mlsanity.com/paper/2405.19598/) (2024)
- ["Are Adversarial Phishing Webpages a Threat in Reality?" Understanding the Users' Perception of Adversarial Webpages](https://mlsanity.com/paper/2404.02832/) (2024)
- [Phishing Detection Leveraging Machine Learning and Deep Learning: A Review](https://mlsanity.com/paper/2205.07411/) (2022)
- [Attacking logo-based phishing website detectors with adversarial perturbations](https://mlsanity.com/paper/2308.09392/) (2023)

## Built on by (most cited first, 7 of 7)

- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [WebGuard++:Interpretable Malicious URL Detection via Bidirectional Fusion of HTML Subgraphs and Multi-Scale Convolutional BERT](https://mlsanity.com/paper/2506.19356/) (2025)
- [Small Language Models for Phishing Website Detection: Cost, Performance, and Privacy Trade-Offs](https://mlsanity.com/paper/2511.15434/) (2025)
- [Role-Aware Multi-modal federated learning system for detecting phishing webpages](https://mlsanity.com/paper/2509.22369/) (2025)
- [Eliciting Least-to-Most Reasoning for Phishing URL Detection](https://mlsanity.com/paper/2601.20270/) (2026)
- [Benchmarking Large Language Models for Zero-shot and Few-shot Phishing URL Detection](https://mlsanity.com/paper/2602.02641/) (2026)
- [The Fragility of Trigger-Tag Mechanisms for Misuse Detection in Open-Weight LLMs](https://mlsanity.com/paper/2610.03124/) (2026)

## Abstract (arXiv)

To address the challenging problem of detecting phishing webpages, researchers have developed numerous solutions, in particular those based on machine learning (ML) algorithms. Among these, brand-based phishing detection that uses models from Computer Vision to detect if a given webpage is imitating a well-known brand has received widespread attention. However, such models are costly and difficult to maintain, as they need to be retrained with labeled dataset that has to be regularly and continuously collected. Besides, they also need to maintain a good reference list of well-known websites and related meta-data for effective performance. In this work, we take steps to study the efficacy of large language models (LLMs), in particular the multimodal LLMs, in detecting phishing webpages. Given that the LLMs are pretrained on a large corpus of data, we aim to make use of their understanding of different aspects of a webpage (logo, theme, favicon, etc.) to identify the brand of a given webpage and compare the identified brand with the domain name in the URL to detect a phishing attack. We propose a two-phase system employing LLMs in both phases: the first phase focuses on brand identification, while the second verifies the domain. We carry out comprehensive evaluations on a newly collected dataset. Our experiments show that the LLM-based system achieves a high detection rate at high precision; importantly, it also provides interpretable evidence for the decisions. Our system also performs significantly better than a state-of-the-art brand-based phishing detection system while demonstrating robustness against two known adversarial attacks.