mlsanityData to 14 Oct 2026

Occam's razor is insufficient to infer the preferences of irrational agents

arXiv 1712.05812 ↗First posted 15 Dec 2017cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

Inverse reinforcement learning (IRL) attempts to infer human rewards or preferences from observed behavior. Since human planning systematically deviates from rationality, several approaches have been tried to account for specific human shortcomings. However, the general problem of inferring the reward function of an agent of unknown rationality has received little attention. Unlike the well-known ambiguity problems in IRL, this one is practically relevant but cannot be resolved by observing the agent's policy in enough environments. This paper shows (1) that a No Free Lunch result implies it is impossible to uniquely decompose a policy into a planning algorithm and reward function, and (2) that even with a reasonable simplicity prior/Occam's razor on the set of decompositions, we cannot distinguish between the true decomposition and others that lead to high regret. To address this, we need simple `normative' assumptions, which cannot be deduced exclusively from observations.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). First posted on arXiv 15 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 0; 2021: 8; 2022: 10; 2023: 9; 2024: 8; 2025: 4; 2026: 4.

45 papers.

Sort:

By area:

  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  5. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  6. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  7. 2021ROAD: The ROad event Awareness Dataset for Autonomous Drivingcited by 54
  8. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  9. 2023Goodhart's Law in Reinforcement Learningcited by 41
  10. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  11. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  12. 2022Misspecification in Inverse Reinforcement Learningcited by 27
33 more
  1. 2023Compositional preference models for aligning LMscited by 26
  2. 2021Meaningful human control: actionable properties for AI system developmentcited by 18
  3. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  4. 2021Impossibility Results in AI: A Surveycited by 13
  5. 2022Recognising the importance of preference change: A call for a coordinated multidisciplinary research effort in the age of AIcited by 12
  6. 2021Uncertain Decisions Facilitate Better Preference Learningcited by 10
  7. 2022Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlookcited by 10
  8. 2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
  9. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
  10. 2023A Bayesian Approach to Robust Inverse Reinforcement Learningcited by 9
  11. 2023Designing Fiduciary Artificial Intelligencecited by 8
  12. 2021Combining Reward Information from Multiple Sourcescited by 7
  13. 2022Perspectives on Incorporating Expert Feedback into Model Updatescited by 7
  14. 2024Value Preferences Estimation and Disambiguation in Hybrid Participatory Systemscited by 7
  15. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrationscited by 6
  16. 2021Toward AI Assistants That Let Designers Designcited by 6
  17. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  18. 2025Agency Is Frame-Dependentcited by 5
  19. 2022Zero-Shot Assistance in Sequential Decision Problemscited by 4
  20. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  21. 2022The dangers in algorithms learning humans' values and irrationalitiescited by 3
  22. 2021Hard Choices and Hard Limits for Artificial Intelligencecited by 2
  23. 2021Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Conceptscited by 2
  24. 2024Cooperation and Control in Delegation Gamescited by 2
  25. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  26. 2023AI Alignment Dialogues: An Interactive Approach to AI Alignment in Support Agentscited by 1
  27. 2024Explaining Learned Reward Functions with Counterfactual Trajectoriescited by 1
  28. 2025An Affective-Taxis Hypothesis for Alignment and Interpretabilitycited by 1
  29. 2026A Mathematical Theory of Value: a synthesis on goal-directed agency under resource constraintscited by 1
  30. 2025Supervised Reward Inferencecited by 0
  31. 2026Fragility of Value under Imperfect Alignmentcited by 0
  32. 2026Math for AI safety: an invitation for mathematicianscited by 0
  33. 2026On the Computational Tractability of Robust Banditscited by 0