# Exposing LLM Vulnerabilities: Adversarial Scam Detection and Performance

arXiv: https://arxiv.org/abs/2412.00621 · HTML: https://mlsanity.com/paper/2412.00621/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment](https://mlsanity.com/paper/2402.14016/) (2024)
- [An LLM can Fool Itself: A Prompt-Based Adversarial Attack](https://mlsanity.com/paper/2310.13345/) (2023)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data](https://mlsanity.com/paper/2106.06410/) (2021)
- [Detecting Scams Using Large Language Models](https://mlsanity.com/paper/2402.03147/) (2024)
- [Combating Phone Scams with LLM-based Detection: Where Do We Stand?](https://mlsanity.com/paper/2409.11643/) (2024)
- [From ML to LLM: Evaluating the Robustness of Phishing Webpage Detection Models against Adversarial Attacks](https://mlsanity.com/paper/2407.20361/) (2024)
- [An Empirical Analysis of SMS Scam Detection Systems](https://mlsanity.com/paper/2210.10451/) (2022)

## Built on by (most cited first, 4 of 4)

- [TeleAntiFraud-28k: An Audio-Text Slow-Thinking Dataset for Telecom Fraud Detection](https://mlsanity.com/paper/2503.24115/) (2025)
- [SAFE-QAQ: End-to-End Slow-Thinking Audio-Text Fraud Detection via Reinforcement Learning](https://mlsanity.com/paper/2601.01392/) (2026)
- [One Size Fits All? A Modular Adaptive Sanitization Kit (MASK) for Customizable Privacy-Preserving Phone Scam Detection](https://mlsanity.com/paper/2510.18493/) (2025)
- [Scam Shield: Multi-Model Voting and Fine-Tuned LLMs Against Adversarial Attacks](https://mlsanity.com/paper/2511.01746/) (2025)

## Abstract (arXiv)

Can we trust Large Language Models (LLMs) to accurately predict scam? This paper investigates the vulnerabilities of LLMs when facing adversarial scam messages for the task of scam detection. We addressed this issue by creating a comprehensive dataset with fine-grained labels of scam messages, including both original and adversarial scam messages. The dataset extended traditional binary classes for the scam detection task into more nuanced scam types. Our analysis showed how adversarial examples took advantage of vulnerabilities of a LLM, leading to high misclassification rate. We evaluated the performance of LLMs on these adversarial scam messages and proposed strategies to improve their robustness.