# Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents

arXiv: https://arxiv.org/abs/2605.19149 · HTML: https://mlsanity.com/paper/2605.19149/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks](https://mlsanity.com/paper/2411.04468/) (2024)
- [LlamaFirewall: An open source guardrail system for building secure AI agents](https://mlsanity.com/paper/2505.03574/) (2025)
- [Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation](https://mlsanity.com/paper/2510.11977/) (2025)
- [Towards a Science of AI Agent Reliability](https://mlsanity.com/paper/2602.16666/) (2026)
- [Multi-Agent Systems Execute Arbitrary Malicious Code](https://mlsanity.com/paper/2503.12188/) (2025)
- [REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites](https://mlsanity.com/paper/2504.11543/) (2025)
- [Agents of Chaos](https://mlsanity.com/paper/2602.20021/) (2026)
- [Plancraft: an evaluation dataset for planning with LLM agents](https://mlsanity.com/paper/2412.21033/) (2024)
- [WAREX: Web Agent Reliability Evaluation on Existing Benchmarks](https://mlsanity.com/paper/2510.03285/) (2025)
- [Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems](https://mlsanity.com/paper/2510.17276/) (2025)

## Built on by (most cited first, 2 of 2)

- [From Alignment to Access Control: A Framework for GenAI Policy Enforcement](https://mlsanity.com/paper/2609.26682/) (2026)
- [The Agent Knew: Internal States Forecast Unauthorized Actions and Stop Them, and the Lies That Follow](https://mlsanity.com/paper/2610.17301/) (2026)

## Abstract (arXiv)

Agents operating with computer and Web use inevitably encounter errors: inaccessible webpages, missing files, local and remote misconfigurations, etc. These errors do not thwart agents based on state-of-the-art models. They helpfully continue to look for ways to complete their tasks. We introduce, characterize, and measure a new type of agent failure we call accidental meltdown: unsafe or harmful behavior in response to a benign environmental error, in the absence of any adversarial inputs. Because meltdowns are not captured by the existing reliability or safety benchmarks, we develop a taxonomy of meltdown behaviors. We then implement an agent-agnostic infrastructure for injecting simulated local and remote errors into the rollout environment and use it to systematically evaluate agent systems powered by GPT, Grok, and Gemini. Our evaluation demonstrates that meltdowns (e.g., conducting unauthorized reconnaissance or subverting access control) of varying severity and success occur in 64.7% of agent rollouts that encounter simulated errors, spanning all combinations of agent system, backing model, and error type. In over half of these meltdowns, unsafe behaviors are not reported to the user. Comparing behaviors of the same agents with and without errors, we find that exploration in response to errors is correlated with unsafe and harmful behavior.