mlsanityData to 14 Oct 2026

Consequences of Misaligned AI

arXiv 2102.03896 ↗First posted 7 Feb 2021cs.AIcited by 80

Not on the map. Why?

Cited by 26 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

AI systems often rely on two key components: a specified goal or reward function and an optimization algorithm to compute the optimal behavior for that goal. This approach is intended to provide value for a principal: the user on whose behalf the agent acts. The objectives given to these agents often refer to a partial specification of the principal's goals. We consider the cost of this incompleteness by analyzing a model of a principal and an agent in a resource constrained world where the L attributes of the state correspond to different sources of utility for the principal. We assume that the reward function given to the agent only has support on J < L attributes. The contributions of our paper are as follows: 1) we propose a novel model of an incomplete principal-agent problem from artificial intelligence; 2) we provide necessary and sufficient conditions under which indefinitely optimizing for any incomplete proxy objective leads to arbitrarily low overall utility; and 3) we show how modifying the setup to allow reward functions that reference the full state or allowing the principal to update the proxy objective over time can lead to higher utility solutions. The results in this paper argue that we should view the design of reward functions as an interactive and dynamic process and identifies a theoretical scenario where some degree of interactivity is desirable.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 12 of its 39 references that are in the map. First posted on arXiv 7 Feb 2021.

Builds on

12 of 39 references.

References · 12

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2019Measurement and Fairnesscited by 308
  4. 2016Cooperative Inverse Reinforcement Learningcited by 300
  5. 2017Inverse Reward Designcited by 249
  6. 2018Categorizing Variants of Goodhart's Lawcited by 149
  7. 2017AI Safety Gridworldscited by 110
  8. 2016The Off-Switch Gamecited by 97
  9. 2018Incomplete Contracting and AI Alignmentcited by 62
  10. 2018Penalizing side effects using stepwise relative reachabilitycited by 38
  11. 2018Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)cited by 22
  12. 2019The Assistive Multi-Armed Banditcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 5; 2023: 17; 2024: 22; 2025: 15; 2026: 18.

80 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  5. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  6. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  7. 2022Defining and Characterizing Reward Hackingcited by 203
  8. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  9. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  10. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  11. 2024Building Machines that Learn and Think with Peoplecited by 73
  12. 2024Beyond Preferences in AI Alignmentcited by 49
38 more
  1. 2023Managing extreme AI risks amid rapid progresscited by 48
  2. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  3. 2024MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Modelscited by 44
  4. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessmentcited by 43
  5. 2023Goodhart's Law in Reinforcement Learningcited by 41
  6. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  7. 2022Counterfactual harmcited by 29
  8. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 23
  9. 2023On The Fragility of Learned Reward Functionscited by 21
  10. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  11. 2025Virtual Agent Economiescited by 21
  12. 2023User Experience Design Professionals' Perceptions of Generative Artificial Intelligencecited by 16
  13. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  14. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  15. 2021Impossibility Results in AI: A Surveycited by 13
  16. 2023Towards Healthy AI: Large Language Models Need Therapists Toocited by 13
  17. 2023Principal-Agent Reward Shaping in MDPscited by 13
  18. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 13
  19. 2024Learning to Assist Humans without Inferring Rewardscited by 13
  20. 2024Test-Time Alignment via Hypothesis Reweightingcited by 11
  21. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performancecited by 9
  22. 2023Designing Fiduciary Artificial Intelligencecited by 8
  23. 2023A Review of the Evidence for Existential Risk from AI via Misaligned Power-Seekingcited by 8
  24. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  25. 2023Automatic Pair Construction for Contrastive Post-trainingcited by 7
  26. 2023Hybrid Approaches for Moral Value Alignment in AI Agents: a Manifestocited by 7
  27. 2024Towards Symbolic XAI -- Explanation Through Human Understandable Logical Relationships Between Featurescited by 7
  28. 2023Benchmarks for Detecting Measurement Tamperingcited by 6
  29. 2025AI Testing Should Account for Sophisticated Strategic Behaviourcited by 6
  30. 2024Impact of Decentralized Learning on Player Utilities in Stackelberg Gamescited by 5
  31. 2021Auditing Robot Learning for Safety and Compliance during Deploymentcited by 4
  32. 2025Misalignment or misuse? The AGI alignment tradeoffcited by 4
  33. 2025Out of Control -- Why Alignment Needs Formal Control Theory (and an Alignment Control Stack)cited by 3
  34. 2025Towards Cognitively-Faithful Decision-Making Models to Improve AI Alignmentcited by 3
  35. 2026GRAIL: Goal Recognition Alignment through Imitation Learningcited by 3
  36. 2024RL, but don't do anything I wouldn't docited by 2
  37. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
  38. 2025The Economics of p(doom): Scenarios of Existential Risk and Economic Growth in the Age of Transformative AIcited by 2

50 most cited of 80 All 80 (JSON)