# Evaluating the Effectiveness and Robustness of Visual Similarity-based Phishing Detection Models

arXiv: https://arxiv.org/abs/2405.19598 · HTML: https://mlsanity.com/paper/2405.19598/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2024.

## Built on by (most cited first, 5 of 5)

- [Multimodal Large Language Models for Phishing Webpage Detection and Identification](https://mlsanity.com/paper/2408.05941/) (2024)
- [MemoPhishAgent: Memory-Augmented Multi-Modal LLM Agent for Phishing URL Detection](https://mlsanity.com/paper/2602.21394/) (2026)
- [Adaptive Linguistic Prompting (ALP) Enhances Phishing Webpage Detection in Multimodal Large Language Models](https://mlsanity.com/paper/2507.13357/) (2025)
- [Clouding the Mirror: Stealthy Prompt Injection Attacks Targeting LLM-based Phishing Detection](https://mlsanity.com/paper/2602.05484/) (2026)
- [TraceScope: Interactive URL Triage via Decoupled Checklist Adjudication](https://mlsanity.com/paper/2604.21840/) (2026)

## Abstract (arXiv)

Phishing attacks pose a significant threat to Internet users, with cybercriminals elaborately replicating the visual appearance of legitimate websites to deceive victims. Visual similarity-based detection systems have emerged as an effective countermeasure, but their effectiveness and robustness in real-world scenarios have been underexplored. In this paper, we comprehensively scrutinize and evaluate the effectiveness and robustness of popular visual similarity-based anti-phishing models using a large-scale dataset of 451k real-world phishing websites. Our analyses of the effectiveness reveal that while certain visual similarity-based models achieve high accuracy on curated datasets in the experimental settings, they exhibit notably low performance on real-world datasets, highlighting the importance of real-world evaluation. Furthermore, we find that the attackers evade the detectors mainly in three ways: (1) directly attacking the model pipelines, (2) mimicking benign logos, and (3) employing relatively simple strategies such as eliminating logos from screenshots. To statistically assess the resilience and robustness of existing models against adversarial attacks, we categorize the strategies attackers employ into visible and perturbation-based manipulations and apply them to website logos. We then evaluate the models' robustness using these adversarial samples. Our findings reveal potential vulnerabilities in several models, emphasizing the need for more robust visual similarity techniques capable of withstanding sophisticated evasion attempts. We provide actionable insights for enhancing the security of phishing defense systems, encouraging proactive actions.