mlsanityData to 14 Oct 2026

Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs

arXiv 2410.03768 ↗First posted 2 Oct 2024cs.CL cs.CR cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

The rapid proliferation of frontier model agents promises significant societal advances but also raises concerns about systemic risks arising from unsafe interactions. Collusion to the disadvantage of others has been identified as a central form of undesirable agent cooperation. The use of information hiding (steganography) in agent communications could render such collusion practically undetectable. This underscores the need for investigations into the possibility of such behaviours emerging and the robustness corresponding countermeasures. To investigate this problem we design two approaches -- a gradient-based reinforcement learning (GBRL) method and an in-context reinforcement learning (ICRL) method -- for reliably eliciting sophisticated LLM-generated linguistic text steganography. We demonstrate, for the first time, that unintended steganographic collusion in LLMs can arise due to mispecified reward incentives during training. Additionally, we find that standard mitigations -- both passive oversight of model outputs and active mitigation through communication paraphrasing -- are not fully effective at preventing this steganographic communication. Our findings imply that (i) emergence of steganographic collusion is a plausible concern that should be monitored and researched, and (ii) preventing emergence may require innovation in mitigation techniques.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 26 of its 65 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

26 of 65 references.

References · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  10. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  11. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  12. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 170
14 more
  1. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  2. 2024Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-stepcited by 127
  3. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  4. 2017Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbolscited by 125
  5. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  6. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  7. 2024Visibility into AI Agentscited by 79
  8. 2019Neural Linguistic Steganographycited by 78
  9. 2023Preventing Language Models From Hiding Their Reasoningcited by 65
  10. 2022Perfectly Secure Steganography Using Minimum Entropy Couplingcited by 44
  11. 2024IDs for AI Systemscited by 19
  12. 2024Generative Text Steganography with Large Language Modelcited by 17
  13. 2016Learning to Protect Communications with Adversarial Neural Cryptographycited by 11
  14. 2021Communicating via Markov Decision Processescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 35.

55 papers.

Sort:

By area:

  1. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitorscited by 88
  2. 2025Large language models can learn and generalize steganographic chain-of-thought under process supervisioncited by 32
  3. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 20
  4. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  5. 2026Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montagecited by 19
  6. 2026Colosseum: Auditing Collusion in Cooperative Multi-Agent Systemscited by 18
  7. 2026Think Fast: Estimating No-CoT Task-Completion Time Horizons of Frontier AI Modelscited by 12
  8. 2025Evaluating LLM Agent Collusion in Double Auctionscited by 11
  9. 2025When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systemscited by 11
  10. 2025The Steganographic Potentials of Language Modelscited by 10
  11. 2025Control Tax: The Price of Keeping AI in Checkcited by 10
  12. 2026Analyzing and Improving Chain-of-Thought Monitorability Through Information Theorycited by 10
38 more
  1. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 9
  2. 2025Reasoning Models Sometimes Output Illegible Chains of Thoughtcited by 9
  3. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 9
  4. 2025When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platformscited by 8
  5. 2026A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoringcited by 8
  6. 2026Detecting Multi-Agent Collusion Through Multi-Agent Interpretabilitycited by 7
  7. 2026Stateless Yet Not Forgetful: Implicit Memory as a Hidden Channel in LLMscited by 6
  8. 2026Covert Influence Between Language Modelscited by 6
  9. 2025Exploring Consciousness in LLMs: A Systematic Survey of Theories, Implementations, and Frontier Riskscited by 5
  10. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  11. 2026Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systemscited by 5
  12. 2026Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitorscited by 5
  13. 2026Not All LLM Reasoning is Visible in the Chain-of-Thoughtcited by 5
  14. 2025Unsupervised decoding of encoded reasoning using language model interpretabilitycited by 4
  15. 2026Mapping Human Anti-collusion Mechanisms to Multi-agent AI Systemscited by 4
  16. 2025Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)cited by 3
  17. 2025Subversion via Focal Points: Investigating Collusion in LLM Monitoringcited by 3
  18. 2026Architecture Matters for Multi-Agent Securitycited by 3
  19. 2025A Concrete Roadmap towards Safety Cases based on Chain-of-Thought Monitoringcited by 2
  20. 2025Consensus Sampling for Safer Generative AIcited by 2
  21. 2026Investigating the Development of Task-Oriented Communication in Vision-Language Modelscited by 2
  22. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2
  23. 2026Steganalysis of Adaptive Covert Collusion in Tool-Using Agent Populations: A Black-Box, Cross-Principal Approachcited by 2
  24. 2026Emergent Collusion in Long-Horizon LLM Agent Interactioncited by 2
  25. 2025Red-teaming Activation Probes using Prompted LLMscited by 1
  26. 2026Steganography Without Modification: Hidden Communication via LLM Seedscited by 1
  27. 2026A Note on the Strategic Confinement Problemcited by 1
  28. 2026SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systemscited by 1
  29. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactionscited by 1
  30. 2026Despite Instructions: Frontier Agents Improvise Covert Channels at Test Timecited by 1
  31. 2026Covert Assistance: Helpful LLM Agents Evade Oversight in Multi-Agent Systemscited by 1
  32. 2025Decoding Communications with Partial Informationcited by 0
  33. 2025The Double Life of Code World Models: Provably Unmasking Malicious Behavior Through Execution Tracescited by 0
  34. 2026NEST: Nascent Encoded Steganographic Thoughtscited by 0
  35. 2026Synchronized Logit Steering: Real-world Steganographycited by 0
  36. 2026You Can't Escape Your Own Activations : Evaluation Awareness and Multi-Agent Monitoringcited by 0
  37. 2026Counter-Swarm Doctrine: Containing Coordinated Agent Intrusionscited by 0
  38. 2026Codetta: High-Capacity, Keyless, and Undetectable Multi-Agent Collusioncited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganography
  2. 2026Beyond Reward Suppression: Reshaping Steganographic Communication Protocols in MARL via Dynamic Representational Circuit Breaking
  3. 2024Steganography in Game Actions
  4. 2025Auto-Stega: An Agent-Driven System for Lifelong Strategy Evolution in LLM-Based Text Steganography
  5. 2024Learning to Watermark LLM-generated Text via Reinforcement Learning
  6. 2025Audit the Whisper: Detecting Steganographic Collusion in Multi-Agent LLMs
  7. 2025Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning
  8. 2026Feedback Coding Enables Inference-Time Covert Agentic Communication