mlsanityData to 14 Oct 2026

Misalignment or misuse? The AGI alignment tradeoff

arXiv 2506.03755 ↗First posted 4 Jun 2025cs.CY cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Creating systems that are aligned with our goals is seen as a leading approach to create safe and beneficial AI in both leading AI companies and the academic field of AI safety. We defend the view that misaligned AGI - future, generally intelligent (robotic) AI agents - poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one another, we show that - in principle - there is room for alignment approaches which do not increase misuse risk. We then investigate how the tradeoff between misalignment and misuse looks empirically for different technical approaches to AI alignment. Here, we argue that many current alignment techniques and foreseeable improvements thereof plausibly increase risks of catastrophic misuse. Since the impacts of AI depend on the social context, we close by discussing important social factors and suggest that to reduce the risk of a misuse catastrophe due to aligned AGI, techniques such as robustness, AI control methods and especially good governance seem essential.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 73 of its 133 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

73 of 133 references.

References · 73

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,642
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  3. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  9. 2020Artificial Intelligence, Values and Alignmentcited by 448
  10. 2019On Evaluating Adversarial Robustnesscited by 447
  11. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
  12. 2024Alignment faking in large language modelscited by 360
  13. 2016Cooperative Inverse Reinforcement Learningcited by 300
  14. 2024Frontier Models are Capable of In-context Schemingcited by 287
  15. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  16. 2023AI Alignment: A Comprehensive Surveycited by 263
  17. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  18. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  19. 2023An Overview of Catastrophic AI Riskscited by 222
  20. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  21. 2023Model evaluation for extreme riskscited by 177
  22. 2021Alignment of Language Agentscited by 168
  23. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  24. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  25. 2023Large Language Models can Strategically Deceive their Users when Put Under Pressurecited by 146
  26. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  27. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  28. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  29. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  30. 2019Optimal Policies Tend to Seek Powercited by 119
  31. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115
  32. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  33. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 87
  34. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  35. 2021Consequences of Misaligned AIcited by 80
  36. 2024Visibility into AI Agentscited by 79
  37. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  38. 2023The Gradient of Generative AI Release: Methods and Considerationscited by 63

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 48
  2. 2026Safety, Security, and Cognitive Risks in World Modelscited by 3
  3. 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviorscited by 3
  4. 2025Against racing to AGI: Cooperation, deterrence, and catastrophic riskscited by 2