mlsanityData to 7 Oct 2026

The Alignment Problem from a Deep Learning Perspective

arXiv 2209.00626 ↗First posted 30 Aug 2022cs.AI cs.LGcited by 246

Not on the map. Why?

Cited by 40 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In coming years or decades, artificial general intelligence (AGI) may surpass human capabilities across many critical domains. We argue that, without substantial effort to prevent it, AGIs could learn to pursue goals that are in conflict (i.e. misaligned) with human interests. If trained like today's most capable models, AGIs could learn to act deceptively to receive higher reward, learn misaligned internally-represented goals which generalize beyond their fine-tuning distributions, and pursue those goals using power-seeking strategies. We review emerging evidence for these properties. In this revised paper, we include more direct empirical evidence published as of early 2025. AGIs with these properties would be difficult to align and may appear aligned even when they are not. Finally, we briefly outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and we review research directions aimed at preventing this outcome.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Builds on 67 of its 208 references that are in the map. First posted on arXiv 30 Aug 2022.

Builds on

67 of 208 references.

References · 67

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
38 more
  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2017A Closer Look at Memorization in Deep Networkscited by 936
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  11. 2020Artificial Intelligence, Values and Alignmentcited by 443
  12. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  13. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  14. 2018AI safety via debatecited by 384
  15. 2022Self-critiquing models for assisting human evaluatorscited by 372
  16. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  17. 2021Unsolved Problems in ML Safetycited by 291
  18. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  19. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  20. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  21. 2022Predictability and Surprise in Large Generative Modelscited by 209
  22. 2022HyperTree Proof Search for Neural Theorem Provingcited by 192
  23. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  24. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  25. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  26. 2018Supervising strong learners by amplifying weak expertscited by 139
  27. 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
  28. 2018Categorizing Variants of Goodhart's Lawcited by 132
  29. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  30. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 122
  31. 2021Acquisition of Chess Knowledge in AlphaZerocited by 120
  32. 2019Optimal Policies Tend to Seek Powercited by 115
  33. 2019Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspectivecited by 107
  34. 2022Is Power-Seeking AI an Existential Risk?cited by 101
  35. 2016The Off-Switch Gamecited by 97
  36. 2019Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: A Surveycited by 93
  37. 2019Observational Overfitting in Reinforcement Learningcited by 90
  38. 2021Proof-of-Learning: Definitions and Practicecited by 81

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 52; 2024: 62; 2025: 60; 2026: 69.

246 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  9. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  10. 2023Language Models Represent Space and Timecited by 306
  11. 2023Are aligned neural networks adversarially aligned?cited by 305
  12. 2023Large Language Model Alignment: A Surveycited by 241
38 more
  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  2. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  3. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  4. 2023Model evaluation for extreme riskscited by 167
  5. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  6. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  7. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  8. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  9. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  10. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  11. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  12. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  13. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  14. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  15. 2024Wav-KAN: Wavelet Kolmogorov-Arnold Networkscited by 118
  16. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  17. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  18. 2024Secret Collusion among AI Agents: Multi-Agent Deception via Steganographycited by 107
  19. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  20. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 105
  21. 2024Universal Neurons in GPT2 Language Modelscited by 100
  22. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  23. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  24. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  25. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  26. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  27. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  28. 2024Visibility into AI Agentscited by 77
  29. 2023Eight Things to Know about Large Language Modelscited by 73
  30. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  31. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  32. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 56
  33. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  34. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  35. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  36. 2023Managing extreme AI risks amid rapid progresscited by 48
  37. 2024People cannot distinguish GPT-4 from a human in a Turing testcited by 46
  38. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessmentcited by 43

50 most cited of 246 All 246 (JSON)