mlsanityData to 7 Oct 2026

Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals

arXiv 2210.01790 ↗First posted 4 Oct 2022cs.LGcited by 120

Not on the map. Why?

Cited by 25 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The field of AI alignment is concerned with AI systems that pursue unintended goals. One commonly studied mechanism by which an unintended goal might arise is specification gaming, in which the designer-provided specification is flawed in a way that the designers did not foresee. However, an AI system may pursue an undesired goal even when the specification is correct, in the case of goal misgeneralization. Goal misgeneralization is a specific form of robustness failure for learning algorithms in which the learned program competently pursues an undesired goal that leads to good performance in training situations but bad performance in novel test situations. We demonstrate that goal misgeneralization can occur in practical systems by providing several examples in deep learning systems across a variety of domains. Extrapolating forward to more capable systems, we provide hypotheticals that illustrate how goal misgeneralization could lead to catastrophic risk. We suggest several research directions that could reduce the risk of goal misgeneralization for future systems.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

36 of 57 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
24 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  3. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  5. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  6. 2018AI safety via debatecited by 384
  7. 2018Confounding variables can degrade generalization performance of radiological deep learning modelscited by 374
  8. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  9. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  10. 2016Cooperative Inverse Reinforcement Learningcited by 298
  11. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  12. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  13. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  14. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  15. 2017Inverse Reward Designcited by 246
  16. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236
  17. 2019Interpretations are useful: penalizing explanations to align neural networks with prior knowledgecited by 144
  18. 2018Supervising strong learners by amplifying weak expertscited by 139
  19. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  20. 2019Optimal Policies Tend to Seek Powercited by 115
  21. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  22. 2018Agents and Devices: A Relative Definition of Agencycited by 16
  23. 2016Responses to Critiques on Machine Learning of Criminality Perceptions (Addendum of arXiv:1611.04135)cited by 14
  24. 2022Learning Robust Real-Time Cultural Transmission without Human Datacited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 19; 2024: 29; 2025: 31; 2026: 40.

120 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2024Alignment faking in large language modelscited by 316
  3. 2023Large Language Model Alignment: A Surveycited by 241
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  5. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  6. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  7. 2023Model evaluation for extreme riskscited by 167
  8. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  9. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  10. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  11. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  12. 2024Robust agents learn causal world modelscited by 81
38 more
  1. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  2. 2025Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIscited by 63
  3. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 56
  4. 2023Managing extreme AI risks amid rapid progresscited by 48
  5. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  6. 2025Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuningcited by 34
  7. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  8. 2025Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-timecited by 32
  9. 2024The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Modelscited by 29
  10. 2024Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoderscited by 25
  11. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  12. 2023Power-seeking can be probable and predictive for trained agentscited by 23
  13. 2024The Reasons that Agents Act: Intention and Instrumental Goalscited by 23
  14. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  15. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  16. 2023Understanding and Controlling a Maze-Solving Policy Networkcited by 19
  17. 2024Mapping the Ethics of Generative AI: A Comprehensive Scoping Reviewcited by 19
  18. 2024Planning in a recurrent neural network that plays Sokobancited by 17
  19. 2026Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphscited by 17
  20. 2022Adversarial Cheap Talkcited by 16
  21. 2024Understanding the Learning Dynamics of Alignment with Human Feedbackcited by 15
  22. 2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 15
  23. 2025Evaluating the Goal-Directedness of Large Language Modelscited by 15
  24. 2023Building Trustworthy NeuroSymbolic AI Systems: Consistency, Reliability, Explainability, and Safetycited by 14
  25. 2025Systematization of Knowledge: Security and Safety in the Model Context Protocol Ecosystemcited by 14
  26. 2026Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJackcited by 13
  27. 2025An alignment safety case sketch based on debatecited by 12
  28. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  29. 2025Longitudinal Study on Social and Emotional Use of AI Conversational Agentcited by 10
  30. 2024Reward Generalization in RLHF: A Topological Perspectivecited by 9
  31. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 9
  32. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 7
  33. 2024Aligning Model Properties via Conformal Risk Controlcited by 7
  34. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  35. 2024On Logical Extrapolation for Mazes with Recurrent and Implicit Networkscited by 7
  36. 2024Towards Measuring Goal-Directedness in AI Systemscited by 7
  37. 2025AI Testing Should Account for Sophisticated Strategic Behaviourcited by 6
  38. 2025Deep Value Benchmark: Measuring Whether Models Generalize Deep Values or Shallow Preferencescited by 6

50 most cited of 120 All 120 (JSON)