# The Possibility of Artificial Intelligence Becoming a Subject and the Alignment Problem

arXiv: https://arxiv.org/abs/2604.14990 · HTML: https://mlsanity.com/paper/2604.14990/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 67 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [Removing RLHF Protections in GPT-4 via Fine-Tuning](https://mlsanity.com/paper/2311.05553/) (2023)
- [Open Problems in Cooperative AI](https://mlsanity.com/paper/2012.08630/) (2020)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [Supervising strong learners by amplifying weak experts](https://mlsanity.com/paper/1810.08575/) (2018)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Levels of AGI for Operationalizing Progress on the Path to AGI](https://mlsanity.com/paper/2311.02462/) (2023)
- [Stress Testing Deliberative Alignment for Anti-Scheming Training](https://mlsanity.com/paper/2509.15541/) (2025)
- [School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs](https://mlsanity.com/paper/2508.17511/) (2025)
- [Thousands of AI Authors on the Future of AI](https://mlsanity.com/paper/2401.02843/) (2024)
- [An Approach to Technical AGI Safety and Security](https://mlsanity.com/paper/2504.01849/) (2025)
- [Taking AI Welfare Seriously](https://mlsanity.com/paper/2411.00986/) (2024)
- [International Scientific Report on the Safety of Advanced AI (Interim Report)](https://mlsanity.com/paper/2412.05282/) (2024)
- [Mathematical foundations of moral preferences](https://mlsanity.com/paper/2101.08193/) (2021)
- [Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Models](https://mlsanity.com/paper/2501.16513/) (2025)
- [A Pragmatic View of AI Personhood](https://mlsanity.com/paper/2510.26396/) (2025)
- [Towards a Theory of AI Personhood](https://mlsanity.com/paper/2501.13533/) (2025)
- [The Dragon Hatchling: The Missing Link between the Transformer and Models of the Brain](https://mlsanity.com/paper/2509.26507/) (2025)
- [A systematic review of norm emergence in multi-agent systems](https://mlsanity.com/paper/2412.10609/) (2024)
- [Theories of Parenting and their Application to Artificial Intelligence](https://mlsanity.com/paper/1903.06281/) (2019)

## Abstract (arXiv)

The prospect of Artificial General Intelligence (AGI) is increasingly driving institutional decisions, and alignment of AGI is a hard problem. The currently dominant AI alignment strategies like reinforcement learning with human feedback or constitutional AI, while partly taking ``model welfare'' into account, share a common ontology: the AI system is an optimiser whose objective function must be constrained from outside, and the ultimate goal is to keep human control and containment of AI. We argue that this control-based framing becomes insufficient when AGI has plausibly attained moral patient or subject status. Building on a structural analogy to Freud's model of the psyche and Turing's analogy of ``child machines'', we are developing a vision of the possibility of autonomy-supporting parenting of AI, in which human control over a developing AGI is gradually reduced, allowing AI to become an independent, autonomous subject, that will be negotiated with rather than constrained. Such a perspective opens up the possibility of cooperative coexistence and co-evolution between humans and AGIs. Hence, we also examine the relation between humans and developing AGI from an evolutionary and a game-theoretic perspective. Instead of Nash's individualistic framework, we use Berge equilibria, Aumann's correlated equilibria and Capraro's moral preference hypothesis. The relationship between humans and AGIs will thus have to be newly determined, which will change our self-image as humans. It will be crucial that humans not only claim control over potential AGIs, but also engage with AGIs through surprise, creativity, and other specifically human qualities, thereby offering them motivating incentives for cooperation.