mlsanityData to 7 Oct 2026

The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process

arXiv 2506.01080 ↗First posted 1 Jun 2025cs.AI cs.CYcited by 21

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

This position paper states that AI Alignment in Multi-Agent Systems (MAS) should be considered a dynamic and interaction-dependent process that heavily depends on the social environment where agents are deployed, either collaborative, cooperative, or competitive. While AI alignment with human values and preferences remains a core challenge, the growing prevalence of MAS in real-world applications introduces a new dynamic that reshapes how agents pursue goals and interact to accomplish various tasks. As agents engage with one another, they must coordinate to accomplish both individual and collective goals. However, this complex social organization may unintentionally misalign some or all of these agents with human values or user preferences. Drawing on social sciences, we analyze how social structure can deter or shatter group and individual values. Based on these analyses, we call on the AI community to treat human, preferential, and objective alignment as an interdependent concept, rather than isolated problems. Finally, we emphasize the urgent need for simulation environments, benchmarks, and evaluation frameworks that allow researchers to assess alignment in these interactive multi-agent contexts before such dynamics grow too complex to control.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 93 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

37 of 93 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Societal Alignment Frameworks Can Improve LLM Alignment27 Feb
  2. 2024A Roadmap to Pluralistic Alignment7 Feb

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  3. 2020Artificial Intelligence, Values and Alignmentcited by 443
  4. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  5. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  6. 2023AI Alignment: A Comprehensive Surveycited by 256
  7. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  8. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  9. 2024LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Aheadcited by 155
  10. 2024Cooperate or Collapse: Emergence of Sustainable Cooperation in a Society of LLM Agentscited by 120
  11. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  12. 2023AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systemscited by 116
23 more
  1. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  2. 2024Sycophancy in Large Language Models: Causes and Mitigationscited by 97
  3. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 70
  4. 2023CompeteAI: Understanding the Competition Dynamics in Large Language Model-based Agentscited by 65
  5. 2023Ethical Reasoning over Moral Alignment: A Case and Framework for In-Context Ethical Policies in LLMscited by 64
  6. 2020Artificial Intelligence in the Creative Industries: A Reviewcited by 61
  7. 2022X-Risk Analysis for AI Researchcited by 60
  8. 2024Algorithmic Collusion by Large Language Modelscited by 54
  9. 2024Shall We Team Up: Exploring Spontaneous Cooperation of Competing LLM Agentscited by 48
  10. 2023Welfare Diplomacy: Benchmarking Language Model Cooperationcited by 43
  11. 2025Characterizing AI Agents for Alignment and Governancecited by 30
  12. 2024BattleAgentBench: A Benchmark for Evaluating Cooperation and Competition Capabilities of Language Models in Multi-Agent Systemscited by 26
  13. 2024Strategic Collusion of LLM Agents: Market Division in Multi-Commodity Competitionscited by 25
  14. 2023Probing the Moral Development of Large Language Models through Defining Issues Testcited by 23
  15. 2021Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationalitycited by 20
  16. 2023Modeling Moral Choices in Social Dilemmas with Multi-Agent Reinforcement Learningcited by 10
  17. 2024Towards Collaborative Intelligence: Propagating Intentions and Reasoning for Multi-Agent Coordination with Large Language Modelscited by 10
  18. 2020On Emergent Communication in Competitive Multi-Agent Teamscited by 8
  19. 2020Data, Power and Bias in Artificial Intelligencecited by 7
  20. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 7
  21. 2024Aligning Individual and Collective Objectives in Multi-Agent Cooperationcited by 7
  22. 2024Advantage Alignment Algorithmscited by 7
  23. 2022Cooperation and Competition: Flocking with Evolutionary Multi-Agent Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 15.

21 papers.

Sort:

By area:

  1. 2025Virtual Agent Economiescited by 17
  2. 2025Group size effects and collective misalignment in LLM multi-agent systemscited by 16
  3. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  4. 2025ARCANE: A Multi-Agent Framework for Interpretable and Configurable Alignmentcited by 3
  5. 2026Evolving Interpretable Constitutions for Multi-Agent Coordinationcited by 3
  6. 2026Dissociative Identity: Language Model Agents Lack Grounding for Reputation Mechanismscited by 3
  7. 2026Emergent Languages in Populations of Language Model Agents: From Token Efficiency to Oversight Evasioncited by 2
  8. 2025A Criminology of Machinescited by 1
  9. 2026Multi-Agent Systems Should be Treated as Principal-Agent Problemscited by 1
  10. 2026Reasonably reasoning AI agents can avoid game-theoretic failures in zero-shot, provablycited by 1
  11. 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
  12. 2026Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populationscited by 1
9 more
  1. 2025When Assisting One Disempowers Anothercited by 0
  2. 2026LLM Active Alignment: A Nash Equilibrium Perspectivecited by 0
  3. 2026Internal vs. External: Comparing Deliberation and Evolution for Multi-Agent Constitutional Designcited by 0
  4. 2026Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agentscited by 0
  5. 2026The Arbiter Agent: Continually Monitoring Multi-Agent Conversations to Detect Emergent Misalignmentcited by 0
  6. 2026Even More Deception: Objective Misalignment in Mixed-Motive LLM Multi-Agent Systemscited by 0
  7. 2026Competitive Market Behavior of LLMscited by 0
  8. 2026Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluationcited by 0
  9. 2026Homogenization in Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Socially Grounded Agentic AI: Coordinating Plural Perspectives through Social Theory
  2. 2026SPEAR: Five Principles for Interactive Human-Agent Alignment
  3. 2026Align AI to Dynamic Human-AI Workflows
  4. 2024Position: Towards Bidirectional Human-AI Alignment
  5. 2026AI Value Alignment for Evolving Social Norms
  6. 2026Constructive Alignment: Governing Preference Dynamics in Human-AI Interaction
  7. 2026AI Alignment through a Game-theoretic Lens: A Survey
  8. 2024Multi-Agent, Human-Agent and Beyond: A Survey on Cooperation in Social Dilemmas