mlsanityData to 7 Oct 2026

Secret Collusion among AI Agents: Multi-Agent Deception via Steganography

arXiv 2402.07510 ↗First posted 12 Feb 2024cs.AI cs.CRcited by 107

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Recent capability increases in large language models (LLMs) open up applications in which groups of communicating generative AI agents solve joint tasks. This poses privacy and security challenges concerning the unauthorised sharing of information, or other unwanted forms of agent coordination. Modern steganographic techniques could render such dynamics hard to detect. In this paper, we comprehensively formalise the problem of secret collusion in systems of generative AI agents by drawing on relevant concepts from both AI and security literature. We study incentives for the use of steganography, and propose a variety of mitigation measures. Our investigations result in a model evaluation framework that systematically tests capabilities required for various forms of secret collusion. We provide extensive empirical results across a range of contemporary LLMs. While the steganographic capabilities of current models remain limited, GPT-4 displays a capability jump suggesting the need for continuous monitoring of steganographic frontier model capabilities. We conclude by laying out a comprehensive research program to mitigate future risks of collusion between generative AI models.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 61 of its 126 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

61 of 126 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 60

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  10. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  11. 2019Machine Unlearningcited by 1,020
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
38 more
  1. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  3. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  4. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  7. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  8. 2021A Practical Guide to Multi-Objective Reinforcement Learning and Planningcited by 240
  9. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 212
  10. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  11. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  12. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  13. 2023On the Reliability of Watermarks for Large Language Modelscited by 170
  14. 2023Birth of a Transformer: A Memory Viewpointcited by 154
  15. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 137
  16. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  17. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  18. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  19. 2023Can Large Language Models Explain Themselves? A Study of LLM-Generated Self-Explanationscited by 90
  20. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  21. 2019Neural Linguistic Steganographycited by 73
  22. 2021Off-Belief Learningcited by 72
  23. 2023Preventing Language Models From Hiding Their Reasoningcited by 62
  24. 2024Can Large Language Model Agents Simulate Human Trust Behavior?cited by 62
  25. 2022Adversarial Training for High-Stakes Reliabilitycited by 60
  26. 2023Privacy Issues in Large Language Models: A Surveycited by 57
  27. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  28. 2022Perfectly Secure Steganography Using Minimum Entropy Couplingcited by 42
  29. 2023Honesty Is the Best Policy: Defining and Mitigating AI Deceptioncited by 40
  30. 2021Mitigating harm in language models with conditional-likelihood filtrationcited by 38
  31. 2018Multi-Agent Common Knowledge Reinforcement Learningcited by 31
  32. 2023Intelligent Virtual Assistants with LLM-based Process Automationcited by 26
  33. 2023Unveiling Theory of Mind in Large Language Models: A Parallel to Single Neurons in the Human Braincited by 18
  34. 2022Equivariant Networks for Zero-Shot Coordinationcited by 16
  35. 2022Illusory Attacks: Information-Theoretic Detectability Matters in Adversarial Attackscited by 12
  36. 2023The Butterfly Effect in Artificial Intelligence Systems: Implications for AI Bias and Fairnesscited by 12
  37. 2021Communicating via Markov Decision Processescited by 9
  38. 2022Learning to Mitigate AI Collusion on Economic Platformscited by 9

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 42; 2026: 65.

107 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  3. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 81
  4. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  5. 2025A Self-Improving Coding Agentcited by 58
  6. 2025Detecting High-Stakes Interactions with Activation Probescited by 45
  7. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 30
  8. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  9. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  10. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  11. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agentscited by 22
  12. 2026A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypescited by 19
38 more
  1. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  2. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montagecited by 17
  3. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  4. 2026Colosseum: Auditing Collusion in Cooperative Multi-Agent Systemscited by 17
  5. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 16
  6. 2025When Ethics and Payoffs Diverge: LLM Agents in Morally Charged Social Dilemmascited by 14
  7. 2025Seven Security Challenges in Cross-domain Multi-agent LLM Systemscited by 12
  8. 2025Enhancing Robustness of LLM-Driven Multi-Agent Systems through Randomized Smoothingcited by 9
  9. 2025When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systemscited by 9
  10. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 9
  11. 2025A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety Riskscited by 8
  12. 2025LLM Applications: Current Paradigms and the Next Frontiercited by 8
  13. 2025Build Agent Advocates, Not Platform Agentscited by 8
  14. 2025Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMscited by 8
  15. 2025Distributional AGI Safetycited by 8
  16. 2026M3-BENCH: Process-Aware Evaluation of LLM Agents' Social Behaviors in Mixed-Motive Gamescited by 8
  17. 2025Many LLMs Are More Utilitarian Than Onecited by 7
  18. 2025The Secret Agenda: LLMs Strategically Lie and Our Current Safety Tools Are Blindcited by 7
  19. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  20. 2025When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platformscited by 7
  21. 2026How are AI agents used? Evidence from 177,000 MCP toolscited by 7
  22. 2025LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactionscited by 6
  23. 2025Toward a Safe Internet of Agentscited by 6
  24. 2026The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societiescited by 6
  25. 2026A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringcited by 6
  26. 2026Detecting Multi-Agent Collusion Through Multi-Agent Interpretabilitycited by 6
  27. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  28. 2025AI Agent Behavioral Sciencecited by 5
  29. 2025Formalizing the Safety, Security, and Functional Properties of Agentic AI Systemscited by 5
  30. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  31. 2026Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systemscited by 5
  32. 2025Scheming Ability in LLM-to-LLM Strategic Interactionscited by 4
  33. 2026Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systemscited by 4
  34. 2026Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agentscited by 4
  35. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 3
  36. 2025Subversion via Focal Points: Investigating Collusion in LLM Monitoringcited by 3
  37. 2025Unsupervised decoding of encoded reasoning using language model interpretabilitycited by 3
  38. 2025Many-to-One Adversarial Consensus: Exposing Multi-Agent Collusion Risks in AI-Based Healthcarecited by 3

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Steganographic Potentials of Language Models
  2. 2026AgentSecBench: Measuring Prompt Injection, Privacy Leakage, and Tool-Use Integrity in LLM Agents
  3. 2025Auto-Stega: An Agent-Driven System for Lifelong Strategy Evolution in LLM-Based Text Steganography
  4. 2026Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs
  5. 2026Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
  6. 2025Automating Prompt Leakage Attacks on Large Language Models Using Agentic Approach
  7. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent
  8. 2026AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems