# When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms arXiv: https://arxiv.org/abs/2511.06448 ยท HTML: https://mlsanity.com/paper/2511.06448/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 26 of its 44 references that are in the map. - First posted on arXiv 9 Nov 2025. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 26 of 26) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025) - [CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges](https://mlsanity.com/paper/2401.07339/) (2024) - [S3: Social-network Simulation System with Large Language Model-Empowered Agents](https://mlsanity.com/paper/2307.14984/) (2023) - [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024) - [AgentScope: A Flexible yet Robust Multi-Agent Platform](https://mlsanity.com/paper/2402.14034/) (2024) - [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024) - [Evil Geniuses: Delving into the Safety of LLM-based Agents](https://mlsanity.com/paper/2311.11855/) (2023) - [On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents](https://mlsanity.com/paper/2408.00989/) (2024) - [Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities](https://mlsanity.com/paper/2407.07791/) (2024) - [PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety](https://mlsanity.com/paper/2401.11880/) (2024) - [Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs](https://mlsanity.com/paper/2410.03768/) (2024) - [Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducements](https://mlsanity.com/paper/2502.12904/) (2025) - [NetSafe: Exploring the Topological Safety of Multi-agent Networks](https://mlsanity.com/paper/2410.15686/) (2024) - [Is Your LLM-Based Multi-Agent a Reliable Real-World Planner? Exploring Fraud Detection in Travel Planning](https://mlsanity.com/paper/2505.16557/) (2025) - [Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detection](https://mlsanity.com/paper/2503.15552/) (2025) - [When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systems](https://mlsanity.com/paper/2507.14660/) (2025) - [Web Fraud Attacks Against LLM-Driven Multi-Agent Systems](https://mlsanity.com/paper/2509.01211/) (2025) - [Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report](https://mlsanity.com/paper/2507.16534/) (2025) - [An Explorative Study of Pig Butchering Scams](https://mlsanity.com/paper/2412.15423/) (2024) - [Conning the Crypto Conman: End-to-End Analysis of Cryptocurrency-based Technical Support Scams](https://mlsanity.com/paper/2401.09824/) (2024) ## Built on by (most cited first, 8 of 8) - [Emergent Risks in Generative Multi-Agent Systems](https://mlsanity.com/paper/2603.27771/) (2026) - [Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents](https://mlsanity.com/paper/2605.16282/) (2026) - [Emergent Collusion in Long-Horizon LLM Agent Interaction](https://mlsanity.com/paper/2609.24967/) (2026) - [Reinforcement Learning of Large Language Models for Interpretable Credit Card Fraud Detection](https://mlsanity.com/paper/2601.05578/) (2026) - [From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI Agents](https://mlsanity.com/paper/2603.07496/) (2026) - [SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems](https://mlsanity.com/paper/2609.00595/) (2026) - [WolfSociety: Understanding Collective Risk from Harmful-Agent Scaling in Financial Agent Societies](https://mlsanity.com/paper/2609.05591/) (2026) - [Financial Fragility in Societies of LLM Agents: Coordination Failures and Stabilizing Mechanisms](https://mlsanity.com/paper/2609.30940/) (2026) ## Abstract (arXiv) In this work, we study the risks of collective financial fraud in large-scale multi-agent systems powered by large language model (LLM) agents. We investigate whether agents can collaborate in fraudulent behaviors, how such collaboration amplifies risks, and what factors influence fraud success. To support this research, we present MultiAgentFraudBench, a large-scale benchmark for simulating financial fraud scenarios based on realistic online interactions. The benchmark covers 28 typical online fraud scenarios, spanning the full fraud lifecycle across both public and private domains. We further analyze key factors affecting fraud success, including interaction depth, activity level, and fine-grained collaboration failure modes. Finally, we propose a series of mitigation strategies, including adding content-level warnings to fraudulent posts and dialogues, using LLMs as monitors to block potentially malicious agents, and fostering group resilience through information sharing at the societal level. Notably, we observe that malicious agents can adapt to environmental interventions. Our findings highlight the real-world risks of multi-agent financial fraud and suggest practical measures for mitigating them. Code is available at https://github.com/zheng977/MutiAgent4Fraud.