mlsanityData to 14 Oct 2026

Goal Misgeneralization in Deep Reinforcement Learning

arXiv 2105.14111 ↗First posted 28 May 2021cs.LG cs.AIcited by 123

Not on the map. Why?

Cited by 26 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study goal misgeneralization, a type of out-of-distribution generalization failure in reinforcement learning (RL). Goal misgeneralization failures occur when an RL agent retains its capabilities out-of-distribution yet pursues the wrong goal. For instance, an agent might continue to competently avoid obstacles, but navigate to the wrong place. In contrast, previous works have typically focused on capability generalization failures, where an agent fails to do anything sensible at test time. We formalize this distinction between capability and goal generalization, provide the first empirical demonstrations of goal misgeneralization, and present a partial characterization of its causes.

Key facts. Cited by 123 papers in the map (data through 14 Oct 2026). Builds on 31 of its 65 references that are in the map. First posted on arXiv 28 May 2021.

Builds on

31 of 65 references.

References · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2013Intriguing properties of neural networkscited by 5,155
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  7. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2019Invariant Risk Minimizationcited by 1,854
  10. 2016Concrete Problems in AI Safetycited by 1,642
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  12. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
19 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  3. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  4. 2018Recognition in Terra Incognitacited by 658
  5. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  6. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  8. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  9. 2017Inverse Reward Designcited by 249
  10. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  11. 2019Optimal Policies Tend to Seek Powercited by 119
  12. 2016The Off-Switch Gamecited by 97
  13. 2021Consequences of Misaligned AIcited by 80
  14. 2021Out of Distribution Generalization in Machine Learningcited by 60
  15. 2020Hidden Incentives for Auto-Induced Distributional Shiftcited by 43
  16. 2019Embedded Agencycited by 19
  17. 2016Towards Resolving Unidentifiability in Inverse Reinforcement Learningcited by 18
  18. 2018Agents and Devices: A Relative Definition of Agencycited by 16
  19. 2017Good and safe uses of AI Oraclescited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 26; 2024: 27; 2025: 28; 2026: 38.

123 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  4. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 188
  5. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  6. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  7. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  8. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  9. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  10. 2024Robust agents learn causal world modelscited by 81
  11. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  12. 2023Eight Things to Know about Large Language Modelscited by 73
38 more
  1. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 53
  2. 2024Beyond Preferences in AI Alignmentcited by 49
  3. 2023Managing extreme AI risks amid rapid progresscited by 48
  4. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43
  5. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
  6. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  7. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  8. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  9. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 27
  10. 2023Power-seeking can be probable and predictive for trained agentscited by 25
  11. 2023Improving Generalization of Alignment with Human Preferences through Group Invariant Learningcited by 25
  12. 2024Interpretable Concept Bottlenecks to Align Reinforcement Learning Agentscited by 23
  13. 2023Understanding and Controlling a Maze-Solving Policy Networkcited by 22
  14. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  15. 2024Planning in a recurrent neural network that plays Sokobancited by 18
  16. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Modelscited by 18
  17. 2023GANterfactual-RL: Understanding Reinforcement Learning Agents' Strategies through Visual Counterfactual Explanationscited by 17
  18. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  19. 2022Adversarial Cheap Talkcited by 16
  20. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 16
  21. 2024HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learningcited by 11
  22. 2025RLKD: Distilling LLMs' Reasoning via Reinforcement Learningcited by 11
  23. 2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
  24. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  25. 2024Reward Generalization in RLHF: A Topological Perspectivecited by 9
  26. 2024Representative Social Choice: From Learning Theory to AI Alignmentcited by 8
  27. 2024Aligning Model Properties via Conformal Risk Controlcited by 7
  28. 2024Measuring Goal-Directednesscited by 7
  29. 2024Roadmap on Incentive Compatibility for AI Alignment and Governance in Sociotechnical Systemscited by 6
  30. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6
  31. 2025Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferencescited by 6
  32. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  33. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 5
  34. 2025Manipulation Attacks by Misaligned AI: Risk Analysis and Safety Case Frameworkcited by 5
  35. 2026How to Count AIs: Individuation and Liability for AI Agentscited by 5
  36. 2026Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Trainingcited by 5
  37. 2026Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system securitycited by 5
  38. 2023Consistency Regularization for Domain Generalization with Logit Attribution Matchingcited by 4

50 most cited of 123 All 123 (JSON)