# The Devil Behind Moltbook: Anthropic Safety is Always Vanishing in Self-Evolving AI Societies

arXiv: https://arxiv.org/abs/2602.09877 · HTML: https://mlsanity.com/paper/2602.09877/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 28 of its 32 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)

## Builds on (most cited first, 28 of 28)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning](https://mlsanity.com/paper/2411.02337/) (2024)
- [Scaling Large Language Model-based Multi-Agent Collaboration](https://mlsanity.com/paper/2406.07155/) (2024)
- [A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systems](https://mlsanity.com/paper/2508.07407/) (2025)
- [EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle](https://mlsanity.com/paper/2510.16079/) (2025)
- [Theory of Mind for Multi-Agent Collaboration via Large Language Models](https://mlsanity.com/paper/2310.10701/) (2023)
- [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior](https://mlsanity.com/paper/2505.16067/) (2025)
- [Multi-Agent Evolve: LLM Self-Improve through Co-evolution](https://mlsanity.com/paper/2510.23595/) (2025)
- [Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agents](https://mlsanity.com/paper/2402.12327/) (2024)
- [Dr. Zero: Self-Evolving Search Agents without Training Data](https://mlsanity.com/paper/2601.07055/) (2026)
- [Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacy](https://mlsanity.com/paper/2407.06813/) (2024)
- [Towards Verifiably Safe Tool Use for LLM Agents](https://mlsanity.com/paper/2601.08012/) (2026)

## Built on by (most cited first, 7 of 7)

- [Agent Behavioral Contracts: Formal Specification and Runtime Enforcement for Reliable Autonomous AI Agents](https://mlsanity.com/paper/2602.22302/) (2026)
- [Social Simulacra in the Wild: AI Agent Communities on Moltbook](https://mlsanity.com/paper/2603.16128/) (2026)
- [Evo-Attacker: Memory-Augmented Reinforcement Learning for Long-Horizon Tool Attacks on LLM-MAS](https://mlsanity.com/paper/2605.25389/) (2026)
- [SAHOO: Safeguarded Alignment for High-Order Optimization Objectives in Recursive Self-Improvement](https://mlsanity.com/paper/2603.06333/) (2026)
- [From Thinker to Society: Security in Hierarchical Autonomy Evolution of AI Agents](https://mlsanity.com/paper/2603.07496/) (2026)
- [How Agents Represent Humans: Human-Directed Stereotypes in an Open Agent Social Network](https://mlsanity.com/paper/2608.22192/) (2026)
- [When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gain](https://mlsanity.com/paper/2609.36535/) (2026)

## Abstract (arXiv)

The emergence of multi-agent systems built from large language models (LLMs) offers a promising paradigm for scalable collective intelligence and self-evolution. Ideally, such systems would achieve continuous self-improvement in a fully closed loop while maintaining robust safety alignment--a combination we term the self-evolution trilemma. However, we demonstrate both theoretically and empirically that an agent society satisfying continuous self-evolution, complete isolation, and safety invariance is impossible. Drawing on an information-theoretic framework, we formalize safety as the divergence degree from anthropic value distributions. We theoretically demonstrate that isolated self-evolution induces statistical blind spots, leading to the irreversible degradation of the system's safety alignment. Empirical and qualitative results from an open-ended agent community (Moltbook) and two closed self-evolving systems reveal phenomena that align with our theoretical prediction of inevitable safety erosion. We further propose several solution directions to alleviate the identified safety concern. Our work establishes a fundamental limit on the self-evolving AI societies and shifts the discourse from symptom-driven safety patches to a principled understanding of intrinsic dynamical risks, highlighting the need for external oversight or novel safety-preserving mechanisms.