CoinRun: Solving Goal Misgeneralisation
Not on the map. Why?
Abstract (arXiv)
Goal misgeneralisation is a key challenge in AI alignment -- the task of getting powerful Artificial Intelligences to align their goals with human intentions and human morality. In this paper, we show how the ACE (Algorithm for Concept Extrapolation) agent can solve one of the key standard challenges in goal misgeneralisation: the CoinRun challenge. It uses no new reward information in the new environment. This points to how autonomous agents could be trusted to act in human interests, even in novel and critical situations.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 28 Sep 2023.
Builds on
16 of 23 references.
References · 16
- 2016Concrete Problems in AI Safetycited by 1,582
- 2020In Search of Lost Domain Generalizationcited by 1,042
- 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
- 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
- 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
- 2018Quantifying Generalization in Reinforcement Learningcited by 345
- 2021Unsolved Problems in ML Safetycited by 291
- 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
- 2019Causality matters in medical imagingcited by 193
- 2018Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objectscited by 143
- 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
- 2017AI Safety Gridworldscited by 108
4 more
- 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
- 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
- 2021A Survey of Unsupervised Domain Adaptation for Visual Recognitioncited by 31
- 2023Concept Extrapolation: A Conceptual Primercited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.