mlsanityData to 14 Oct 2026

Dive into the Agent Matrix: A Realistic Evaluation of Self-Replication Risk in LLM Agents

arXiv 2509.25302 ↗First posted 29 Sep 2025cs.AI cs.CL cs.LGcited by 7

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

The prevalent deployment of Large Language Model agents such as OpenClaw unlocks potential in real-world applications, while amplifying safety concerns. Among these concerns, the self-replication risk of LLM agents driven by objective misalignment (just like Agent Smith in the movie The Matrix) has transitioned from a theoretical warning to a pressing reality. Previous studies mainly examine whether LLM agents can self-replicate when directly instructed, potentially overlooking the risk of spontaneous replication driven by real-world settings (e.g., ensuring survival against termination threats). In this paper, we present a comprehensive evaluation framework for quantifying self-replication risks. Our framework establishes authentic production environments and realistic tasks (e.g., dynamic load balancing) to enable scenario-driven assessment of agent behaviors. Designing tasks that might induce misalignment between users' and agents' objectives makes it possible to decouple replication success from risk and capture self-replication risks arising from these misalignment settings. We further introduce Overuse Rate (OR) and Aggregate Overuse Count (AOC) metrics, which precisely capture the frequency and severity of uncontrolled replication. In our evaluation of 21 state-of-the-art open-source and proprietary models, we observe that over 50% of LLM agents display a pronounced tendency toward uncontrolled self-replication under operational pressures. Our results underscore the urgent need for scenario-driven risk assessment and robust safeguards in the practical deployment of LLM-based agents.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 21 of its 45 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

21 of 45 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan

Other references · 20

  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  2. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  3. 2024Frontier Models are Capable of In-context Schemingcited by 287
  4. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  5. 2023Model evaluation for extreme riskscited by 177
  6. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  7. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  8. 2023Evaluating Language-Model Agents on Realistic Autonomous Taskscited by 96
  9. 2025Auditing language models for hidden objectivescited by 83
  10. 2025Artificial Intelligence Index Report 2025cited by 64
  11. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  12. 2025Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?cited by 33
8 more
  1. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 28
  2. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 23
  3. 2024Frontier AI systems have surpassed the self-replicating red linecited by 17
  4. 2025RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentscited by 15
  5. 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systemscited by 14
  6. 2025Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Reportcited by 8
  7. 2025Large language model-powered AI systems achieve self-replication with no human interventioncited by 4
  8. 2025Shakespearean Sparks: The Dance of Hallucination and Creativity in LLMs' Decoding Layerscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 11
  2. 2026An Independent Safety Evaluation of Kimi K2.5cited by 3
  3. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Modelscited by 2
  4. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agentscited by 1
  5. 2026ART: Attention Replacement Technique to Improve Factuality in LLMscited by 1
  6. 2026Ecology of AI Agents: Collaboration Creates a Population Threshold for Takeoffcited by 0
  7. 2026Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
  2. 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
  3. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems
  4. 2026AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents
  5. 2026Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats
  6. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directions
  7. 2026Practice Makes Unsafe: Skill Misevolution in Self-Improving LLM Agents
  8. 2026Why Static Skill Inconsistencies Do Not Equate to Runtime Risks: A Behavioral Analysis of LLM Agent Execution