# A Roadmap for Robust End-to-End Alignment

arXiv: https://arxiv.org/abs/1809.01036 · HTML: https://mlsanity.com/paper/1809.01036/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 27 of its 112 references that are in the map.
- First posted on arXiv 4 Sep 2018.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Towards A Rigorous Science of Interpretable Machine Learning](https://mlsanity.com/paper/1702.08608/) (2017)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [One pixel attack for fooling deep neural networks](https://mlsanity.com/paper/1710.08864/) (2017)
- [The Hidden Vulnerability of Distributed Learning in Byzantium](https://mlsanity.com/paper/1802.07927/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [Categorizing Variants of Goodhart's Law](https://mlsanity.com/paper/1803.04585/) (2018)
- [A Dual Approach to Scalable Verification of Deep Networks](https://mlsanity.com/paper/1803.06567/) (2018)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Research Priorities for Robust and Beneficial Artificial Intelligence](https://mlsanity.com/paper/1602.03506/) (2016)
- [A Voting-Based System for Ethical Decision Making](https://mlsanity.com/paper/1709.06692/) (2017)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [When Will AI Exceed Human Performance? Evidence from AI Experts](https://mlsanity.com/paper/1705.08807/) (2017)
- [Pragmatic-Pedagogic Value Alignment](https://mlsanity.com/paper/1707.06354/) (2017)
- [A Game-Based Approximate Verification of Deep Neural Networks with Provable Guarantees](https://mlsanity.com/paper/1807.03571/) (2018)
- [Context-Aware Generative Adversarial Privacy](https://mlsanity.com/paper/1710.09549/) (2017)
- [Death and Suicide in Universal Artificial Intelligence](https://mlsanity.com/paper/1606.00652/) (2016)
- [Asynchronous Byzantine Machine Learning (the case of SGD)](https://mlsanity.com/paper/1802.07928/) (2018)
- [Dynamic Safe Interruptibility for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1704.02882/) (2017)

## Abstract (arXiv)

This paper discussed the it robust alignment problem, that is, the problem of aligning the goals of algorithms with human preferences. It presented a general roadmap to tackle this issue. Interestingly, this roadmap identifies 5 critical steps, as well as many relevant aspects of these 5 steps. In other words, we have presented a large number of hopefully more tractable subproblems that readers are highly encouraged to tackle. Hopefully, this combination allows to better highlight the most pressing problems, how every expertise can be best used to, and how combining the solutions to subproblems might add up to solve robust alignment.