# CoinRun: Solving Goal Misgeneralisation

arXiv: https://arxiv.org/abs/2309.16166 · HTML: https://mlsanity.com/paper/2309.16166/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [The Pitfalls of Simplicity Bias in Neural Networks](https://mlsanity.com/paper/2006.07710/) (2020)
- [Unsolved Problems in ML Safety](https://mlsanity.com/paper/2109.13916/) (2021)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Causality matters in medical imaging](https://mlsanity.com/paper/1912.08142/) (2019)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [AI Safety Gridworlds](https://mlsanity.com/paper/1711.09883/) (2017)
- [Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization](https://mlsanity.com/paper/2105.05612/) (2021)
- [Diversify and Disambiguate: Learning From Underspecified Data](https://mlsanity.com/paper/2202.03418/) (2022)
- [A Survey of Unsupervised Domain Adaptation for Visual Recognition](https://mlsanity.com/paper/2112.06745/) (2021)
- [Concept Extrapolation: A Conceptual Primer](https://mlsanity.com/paper/2306.10999/) (2023)

## Built on by (most cited first, 1 of 1)

- [Inoculation Adapters: Improved Selective Generalization of Capabilities with Fewer Surprising Backdoors](https://mlsanity.com/paper/2606.30252/) (2026)

## Abstract (arXiv)

Goal misgeneralisation is a key challenge in AI alignment -- the task of getting powerful Artificial Intelligences to align their goals with human intentions and human morality. In this paper, we show how the ACE (Algorithm for Concept Extrapolation) agent can solve one of the key standard challenges in goal misgeneralisation: the CoinRun challenge. It uses no new reward information in the new environment. This points to how autonomous agents could be trusted to act in human interests, even in novel and critical situations.