# Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)

arXiv: https://arxiv.org/abs/2506.17846 · HTML: https://mlsanity.com/paper/2506.17846/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 21 Jun 2025.

## Builds on (most cited first, 30 of 36)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Robust Adversarial Reinforcement Learning](https://mlsanity.com/paper/1703.02702/) (2017)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [The Alignment Problem from a Deep Learning Perspective](https://mlsanity.com/paper/2209.00626/) (2022)
- [AI Control: Improving Safety Despite Intentional Subversion](https://mlsanity.com/paper/2312.06942/) (2023)
- [Measuring Progress on Scalable Oversight for Large Language Models](https://mlsanity.com/paper/2211.03540/) (2022)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Optimal Policies Tend to Seek Power](https://mlsanity.com/paper/1912.01683/) (2019)
- [Truthful AI: Developing and governing AI that does not lie](https://mlsanity.com/paper/2110.06674/) (2021)
- [Is Power-Seeking AI an Existential Risk?](https://mlsanity.com/paper/2206.13353/) (2022)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Consequences of Misaligned AI](https://mlsanity.com/paper/2102.03896/) (2021)
- [Neural Linguistic Steganography](https://mlsanity.com/paper/1909.01496/) (2019)
- [CycleGAN, a Master of Steganography](https://mlsanity.com/paper/1712.02950/) (2017)

## Built on by (most cited first, 3 of 3)

- [Typed Chain-of-Thought: A Curry-Howard Framework for Verifying LLM Reasoning](https://mlsanity.com/paper/2510.01069/) (2025)
- [Towards Measurement Theory for Artificial Intelligence](https://mlsanity.com/paper/2507.05587/) (2025)
- [Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirements](https://mlsanity.com/paper/2605.12963/) (2026)

## Abstract (arXiv)

This position paper argues that formal optimal control theory should be central to AI alignment research, offering a distinct perspective from prevailing AI safety and security approaches. While recent work in AI safety and mechanistic interpretability has advanced formal methods for alignment, they often fall short of the generalisation required of control frameworks for other technologies. There is also a lack of research into how to render different alignment/control protocols interoperable. We argue that by recasting alignment through principles of formal optimal control and framing alignment in terms of hierarchical stack from physical to socio-technical layers according to which controls may be applied we can develop a better understanding of the potential and limitations for controlling frontier models and agentic AI systems. To this end, we introduce an Alignment Control Stack which sets out a hierarchical layered alignment stack, identifying measurement and control characteristics at each layer and how different layers are formally interoperable. We argue that such analysis is also key to the assurances that will be needed by governments and regulators in order to see AI technologies sustainably benefit the community. Our position is that doing so will bridge the well-established and empirically validated methods of optimal control with practical deployment considerations to create a more comprehensive alignment framework, enhancing how we approach safety and reliability for advanced AI systems.