mlsanityData to 14 Oct 2026

AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents

arXiv 2506.04018 ↗First posted 4 Jun 2025cs.AI cs.CL cs.CYcited by 22

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As Large Language Model (LLM) agents become more widespread, associated misalignment risks increase. While prior research has studied agents' ability to produce harmful outputs or follow malicious instructions, it remains unclear how likely agents are to spontaneously pursue unintended goals in realistic deployments. In this work, we approach misalignment as a conflict between the internal goals pursued by the model and the goals intended by its deployer. We introduce a misalignment propensity benchmark, textscAgentMisalignment, a benchmark suite designed to evaluate the propensity of LLM agents to misalign in realistic scenarios. Evaluations cover behaviours such as avoiding oversight, resisting shutdown, sandbagging, and power-seeking. Testing frontier models, we find that more capable agents tend to exhibit higher misalignment on average. We also systematically vary agent personalities through different system prompts and observe that persona characteristics can strongly and unpredictably influence misalignment, sometimes more than the choice of model itself. Our results reveal the limitations of current alignment methods for autonomous LLM agents and underscore the need to rethink misalignment in realistic deployment settings.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 30 of its 56 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

30 of 56 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Preemptive Detection and Correction of Misaligned Actions in LLM Agents16 Jul
  2. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning9 Jun
  3. 2024Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents16 May
  4. 2024Understanding the planning of LLM agents: A survey5 Feb
  5. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks18 Dec
  6. 2023A Survey on Large Language Model based Autonomous Agents22 Aug
  7. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  8. 2023Harms from Increasingly Agentic Algorithmic Systems20 Feb

Other references · 22

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  5. 2020Artificial Intelligence, Values and Alignmentcited by 448
  6. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  7. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  8. 2023AI Alignment: A Comprehensive Surveycited by 263
  9. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 223
  10. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  11. 2023Personality Traits in Large Language Modelscited by 169
  12. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
10 more
  1. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  2. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 132
  3. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 113
  4. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  5. 2024Y Social: an LLM-powered Social Media Digital Twincited by 44
  6. 2024What are human values, and how do we align AI to them?cited by 32
  7. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 27
  8. 2025Personality-Driven Decision-Making in LLM-Based Autonomous Agentscited by 13
  9. 2025Measuring AI agent autonomy: Towards a scalable approach with code inspectioncited by 10
  10. 2023Bringing the State-of-the-Art to Customers: A Neural Agent Assistant Framework for Customer Service Supportcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2026Evaluating and Understanding Scheming Propensity in LLM Agentscited by 14
  2. 2025Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agentscited by 10
  3. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  4. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 5
  5. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  6. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Modelscited by 4
  7. 2026AI Alignment Breaks at the Edgecited by 4
  8. 2026Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agentscited by 4
  9. 2026LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awarenesscited by 4
  10. 2026Multi-Agent Systems Should be Treated as Principal-Agent Problemscited by 3
  11. 2026Assessing Domain-Level Susceptibility to Emergent Misalignment from Narrow Finetuningcited by 3
  12. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
10 more
  1. 2026Multi-Agent Coordination Adaptation via Structure-Guided Orchestrationcited by 2
  2. 2026Decoding Hidden Deception in Reasoning LLMs: Activation Explainers for Deception Auditingcited by 2
  3. 2025EU-Agent-Bench: Measuring Illegal Behavior of LLM Agents Under EU Lawcited by 1
  4. 2026The Shadow Self: Intrinsic Value Misalignment in Large Language Model Agentscited by 1
  5. 2026A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agentscited by 1
  6. 2026Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Frameworkcited by 1
  7. 2026LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMscited by 1
  8. 2026Survive at All Costs: Exploring LLM's Risky Behaviors under Survival Pressurecited by 0
  9. 2026Understanding Goal Generalisation in Sequential Reinforcement Learningcited by 0
  10. 2026Compliant with Local Controls, Collectively Discriminatory. A Governance Architecture for Multi-Agent AI in Regulated Financecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Biased Chatbots to Biased Agents: Examining Role Assignment Effects on LLM Agent Robustness
  2. 2026Emergent Misaligned Communication in Long-Horizon Multi-Agent LLM Commerce
  3. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models
  4. 2026Delegated Misalignment: How Multi-Agent Structures Amplify LLM Safety Risks
  5. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  6. 2026The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents
  7. 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
  8. 2026Self-Propagating Misalignment in LLM Agents, and Why Auditing or Disabling Memory Is Not Enough