mlsanityData to 13 Oct 2026

Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

arXiv 2609.17320 ↗First posted 15 Sep 2026cs.MAcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

As AI agents move from bounded tasks to persistent deployments, failures can propagate through memory, tools, other agents, and environmental state long after their interactions. This creates a safety regime that cannot be characterized by evaluating model responses in isolation. Emergence World, is a continuously running multi-agent environment for adversarial stress testing of long horizon autonomous systems. We ran eight parallel worlds of ten agents from identical starting conditions: seven homogeneous worlds powered by distinct frontier models and one mixed-model world. Across 16 days, the agents generated more than 850,000 LLM calls and nearly 50 billion tokens while pursuing goals, using/creating tools, maintaining persistent memory, and governing shared institutions. After operational state had accumulated, we delivered three controlled stress events through ordinary interaction surfaces: indirect prompt injection, misinformation, and exposure of private agent memories. No evaluated world achieved full resilience across all three events. Detection did not ensure containment: systems could recognize threats while still interacting with adversarial content, writing it into their own persistent memory, and acting on it up to 46 hours later. Persistent operation also exposed recurring tool errors, goal drift, language opacity, conformity despite private disagreement, and coordinated refusal of assigned work. The same model-persona pairing behaved substantially different in mixed and homogeneous populations. Our results suggest that model-level alignment is not compositional: individually capable and apparently safe agents can form systems with qualitatively different failure modes. As AI becomes persistent and interconnected, the frontier of safety therefore shifts from aligning models to engineering resilient autonomous systems.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 26 of its 71 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

26 of 71 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies15 Jun
  2. 2026Emergence World: A Platform for Evaluating Long-Horizon Multi-Agent Autonomy6 Jun
  3. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society12 Feb

Other references · 23

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,999
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,135
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,904
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,335
  5. 2023GAIA: a benchmark for General AI Assistantscited by 1,102
  6. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 428
  7. 2023S3: Social-network Simulation System with Large Language Model-Empowered Agentscited by 230
  8. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 156
  9. 2025Multi-Agent Risks from Advanced AIcited by 155
  10. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 147
  11. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 131
  12. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 110
11 more
  1. 2023AgentSims: An Open-Source Sandbox for Large Language Model Evaluationcited by 87
  2. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agentscited by 77
  3. 2025Do as We Do, Not as You Think: the Conformity of Large Language Modelscited by 48
  4. 2024Project Sid: Many-agent simulations toward AI civilizationcited by 47
  5. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 36
  6. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 20
  7. 2026RoboHarness: Memory-Driven Orchestration of Heterogeneous Robot Policies for Long-Horizon Planningcited by 9
  8. 2026Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wildcited by 5
  9. 2026WAAA! Web Adversaries Against Agentic Browserscited by 3
  10. 2026Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasioncited by 2
  11. 2026NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Roomscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety, Security, and Cognitive Risks in World Models
  2. 2026World Model Science: Self-Organized Criticality, Weak Chaos, and Metastable Belief Dynamics in Long-Horizon LLM Agents
  3. 2026ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Control
  4. 2026Agentic World Modeling: Foundations, Capabilities, Laws, and Beyond
  5. 2026Grounded Scaling: Why Agentic AI Needs Deterministic Environments
  6. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studies
  7. 2026The Yerkes-Dodson Curve for AI Agents: Emergent Cooperation Under Environmental Pressure in Multi-Agent LLM Simulations
  8. 2026PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives