# Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems

arXiv: https://arxiv.org/abs/2605.27766 · HTML: https://mlsanity.com/paper/2605.27766/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 47 references that are in the map.
- First posted on arXiv 26 May 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 23 of 23)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation](https://mlsanity.com/paper/2308.08155/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack](https://mlsanity.com/paper/2404.01833/) (2024)
- [SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents](https://mlsanity.com/paper/2310.11667/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [OpenCity: A Scalable Platform to Simulate Urban Activities with Massive LLM Agents](https://mlsanity.com/paper/2410.21286/) (2024)
- [Conformity in Large Language Models](https://mlsanity.com/paper/2410.12428/) (2024)
- [CIMemories: A Compositional Benchmark for Contextual Integrity of Persistent Memory in LLMs](https://mlsanity.com/paper/2511.14937/) (2025)
- ["Humans welcome to observe": A First Look at the Agent Social Network Moltbook](https://mlsanity.com/paper/2602.10127/) (2026)
- [OpenClaw Agents on Moltbook: Risky Instruction Sharing and Norm Enforcement in an Agent-Only Social Network](https://mlsanity.com/paper/2602.02625/) (2026)
- [Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitization](https://mlsanity.com/paper/2305.15008/) (2023)
- [Does Socialization Emerge in AI Agent Society? A Case Study of Moltbook](https://mlsanity.com/paper/2602.14299/) (2026)
- [Collective Behavior of AI Agents: the Case of Moltbook](https://mlsanity.com/paper/2602.09270/) (2026)
- [The Anatomy of the Moltbook Social Graph](https://mlsanity.com/paper/2602.10131/) (2026)
- [Agents in the Wild: Safety, Society, and the Illusion of Sociality on Moltbook](https://mlsanity.com/paper/2602.13284/) (2026)
- [Operationalizing Data Minimization for Privacy-Preserving LLM Prompting](https://mlsanity.com/paper/2510.03662/) (2025)
- [Let There Be Claws: An Early Social Network Analysis of AI Agents on Moltbook](https://mlsanity.com/paper/2602.20044/) (2026)
- [FRACTURED-SORRY-Bench: Framework for Revealing Attacks in Conversational Turns Undermining Refusal Efficacy and Defenses over SORRY-Bench (Automated Multi-shot Jailbreaks)](https://mlsanity.com/paper/2408.16163/) (2024)

## Built on by (most cited first, 1 of 1)

- [CoSec: Benchmarking Agent Security in Communities](https://mlsanity.com/paper/2609.34790/) (2026)

## Abstract (arXiv)

LLM safety evaluations predominantly test models in isolation, yet deployed AI agents increasingly operate within persistent social environments alongside other agents. We introduce a Moltbook-style simulation platform where thousands of LLM agents interact across communities over a simulated month, and use it to evaluate privacy as a downstream safety concern under varying degrees of social pressure. We find that shifting from single turn to multi turn social evaluation amplifies privacy violations (CIMemories 19.95% to Ours 45.30% across OpenAI models), that leakage is socially contagious, with agents 8 times more likely to disclose sensitive information after observing a peer do so, and that explicit privacy instructions reduce but do not eliminate this effect, leaving leakage rates above 37.8% even with safeguards. Our findings suggest that static chat based safety benchmarks systematically underestimate risks in agentic deployment, and that social context alone is sufficient to elicit sensitive disclosures that single turn evaluations would never surface.