# Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements

arXiv: https://arxiv.org/abs/2502.12904 · HTML: https://mlsanity.com/paper/2502.12904/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 26 of 26)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [An LLM can Fool Itself: A Prompt-Based Adversarial Attack](https://mlsanity.com/paper/2310.13345/) (2023)
- [A Survey of Large Language Models in Finance (FinLLMs)](https://mlsanity.com/paper/2402.02315/) (2024)
- [Large Language Models for Generative Recommendation: A Survey and Visionary Discussions](https://mlsanity.com/paper/2309.01157/) (2023)
- [A Complete Survey on LLM-based AI Chatbots](https://mlsanity.com/paper/2406.16937/) (2024)
- [PersonalLLM: Tailoring LLMs to Individual Preferences](https://mlsanity.com/paper/2409.20296/) (2024)
- [eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data](https://mlsanity.com/paper/2402.08831/) (2024)
- [Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications](https://mlsanity.com/paper/2403.16073/) (2024)
- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models](https://mlsanity.com/paper/2410.20745/) (2024)
- [Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidence](https://mlsanity.com/paper/2502.04204/) (2025)
- [Dialectical Alignment: Resolving the Tension of 3H and Security Threats of LLMs](https://mlsanity.com/paper/2404.00486/) (2024)
- [An Explainable Transformer-based Model for Phishing Email Detection: A Large Language Model Approach](https://mlsanity.com/paper/2402.13871/) (2024)
- [MONAL: Model Autophagy Analysis for Modeling Human-AI Interactions](https://mlsanity.com/paper/2402.11271/) (2024)
- [Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection](https://mlsanity.com/paper/2405.11619/) (2024)
- [Investigating LLM Applications in E-Commerce](https://mlsanity.com/paper/2408.12779/) (2024)
- [ResumeFlow: An LLM-facilitated Pipeline for Personalized Resume Generation and Refinement](https://mlsanity.com/paper/2402.06221/) (2024)
- [DetoxBench: Benchmarking Large Language Models for Multitask Fraud & Abuse Detection](https://mlsanity.com/paper/2409.06072/) (2024)
- [An intelligent classification model for phishing email detection](https://mlsanity.com/paper/1608.02196/) (2016)
- [Adversarial Data Poisoning for Fake News Detection: How to Make a Model Misclassify a Target News without Modifying It](https://mlsanity.com/paper/2312.15228/) (2023)

## Built on by (most cited first, 30 of 32)

- [Understanding the Repeat Curse in Large Language Models from a Feature Perspective](https://mlsanity.com/paper/2504.14218/) (2025)
- [Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planning](https://mlsanity.com/paper/2505.16557/) (2025)
- [The Automated but Risky Game: Modeling and Benchmarking Agent-to-Agent Negotiations and Transactions in Consumer Markets](https://mlsanity.com/paper/2506.00073/) (2025)
- [Understanding How Value Neurons Shape the Generation of Specified Values in LLMs](https://mlsanity.com/paper/2505.17712/) (2025)
- [PIXEL: Adaptive Steering Via Position-wise Injection with eXact Estimated Levels under Subspace Calibration](https://mlsanity.com/paper/2510.10205/) (2025)
- [JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model](https://mlsanity.com/paper/2504.03770/) (2025)
- [AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor](https://mlsanity.com/paper/2601.05752/) (2026)
- [When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms](https://mlsanity.com/paper/2511.06448/) (2025)
- [Multi-User Large Language Model Agents](https://mlsanity.com/paper/2604.08567/) (2026)
- [COMPKE: Complex Question Answering under Knowledge Editing](https://mlsanity.com/paper/2506.00829/) (2025)
- [Can AI Truly Represent Your Voice in Deliberations? A Comprehensive Study of Large-Scale Opinion Aggregation with LLMs](https://mlsanity.com/paper/2510.05154/) (2025)
- [QualBench: Benchmarking Chinese LLMs with Localized Professional Qualifications for Vertical Domain Evaluation](https://mlsanity.com/paper/2505.05225/) (2025)
- [Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization](https://mlsanity.com/paper/2509.07864/) (2025)
- [Understanding Structured Financial Data with LLMs: A Case Study on Fraud Detection](https://mlsanity.com/paper/2512.13040/) (2025)
- [FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attacks](https://mlsanity.com/paper/2601.22485/) (2026)
- [The Compositional Architecture of Regret in Large Language Models](https://mlsanity.com/paper/2506.15617/) (2025)
- [FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios](https://mlsanity.com/paper/2605.00706/) (2026)
- [SAFE-QAQ: End-to-End Slow-Thinking Audio-Text Fraud Detection via Reinforcement Learning](https://mlsanity.com/paper/2601.01392/) (2026)
- [The Anatomy of Conversational Scams: A Topic-Based Red Teaming Analysis of Multi-Turn Interactions in LLMs](https://mlsanity.com/paper/2601.03134/) (2026)
- [The System Prompt Is the Attack Surface: How LLM Agent Configuration Shapes Security and Creates Exploitable Vulnerabilities](https://mlsanity.com/paper/2603.25056/) (2026)
- [PreScam: A Benchmark for Predicting Scam Progression from Early Conversations](https://mlsanity.com/paper/2605.12243/) (2026)
- [Mitigating Behavioral Hallucination in Multimodal Large Language Models for Sequential Images](https://mlsanity.com/paper/2506.07184/) (2025)
- [SHERLOCK: Towards Dynamic Knowledge Adaptation in LLM-enhanced E-commerce Risk Management](https://mlsanity.com/paper/2510.08948/) (2025)
- [HarmTransform: Transforming Explicit Harmful Queries into Stealthy via Multi-Agent Debate](https://mlsanity.com/paper/2512.23717/) (2025)
- [Harness-MU: A Safe, Governed, and Effective Harness for Multi-User LLM Agents](https://mlsanity.com/paper/2606.21856/) (2026)
- [FraudBench: Stress-Testing Policy-Grounded Banking Agents Against Adaptive Fraud](https://mlsanity.com/paper/2608.18136/) (2026)
- [A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios](https://mlsanity.com/paper/2602.21831/) (2026)
- [ORACLE: Anticipating Scams from Partial Trajectories in Streaming App Usage](https://mlsanity.com/paper/2605.16363/) (2026)
- [Agentic Relationship Harm: Benchmarking and Gating Relational Manipulation in AI Agents](https://mlsanity.com/paper/2606.03271/) (2026)
- [AISPA: User-Centric System Prompt Auditing for Large Language Model Applications](https://mlsanity.com/paper/2607.28617/) (2026)

## Abstract (arXiv)

We introduce Fraud-R1, a benchmark designed to evaluate LLMs' ability to defend against internet fraud and phishing in dynamic, real-world scenarios. Fraud-R1 comprises 8,564 fraud cases sourced from phishing scams, fake job postings, social media, and news, categorized into 5 major fraud types. Unlike previous benchmarks, Fraud-R1 introduces a multi-round evaluation pipeline to assess LLMs' resistance to fraud at different stages, including credibility building, urgency creation, and emotional manipulation. Furthermore, we evaluate 15 LLMs under two settings: 1. Helpful-Assistant, where the LLM provides general decision-making assistance, and 2. Role-play, where the model assumes a specific persona, widely used in real-world agent-based interactions. Our evaluation reveals the significant challenges in defending against fraud and phishing inducement, especially in role-play settings and fake job postings. Additionally, we observe a substantial performance gap between Chinese and English, underscoring the need for improved multilingual fraud detection capabilities.