mlsanityData to 14 Oct 2026

Learning the Preferences of Ignorant, Inconsistent Agents

arXiv 1512.05832 ↗First posted 18 Dec 2015cs.AIcited by 45

Not on the map. Why?

Abstract (arXiv)

An important use of machine learning is to learn what people value. What posts or photos should a user be shown? Which jobs or activities would a person find rewarding? In each case, observations of people's past choices can inform our inferences about their likes and preferences. If we assume that choices are approximately optimal according to some utility function, we can treat preference inference as Bayesian inverse planning. That is, given a prior on utility functions and some observed choices, we invert an optimal decision-making process to infer a posterior distribution on utility functions. However, people often deviate from approximate optimality. They have false beliefs, their planning is sub-optimal, and their choices may be temporally inconsistent due to hyperbolic discounting and other biases. We demonstrate how to incorporate these deviations into algorithms for preference inference by constructing generative models of planning for agents who are subject to false beliefs and time inconsistency. We explore the inferences these models make about preferences, beliefs, and biases. We present a behavioral experiment in which human subjects perform preference inference given the same observations of choices as our model. Results show that human subjects (like our model) explain choices in terms of systematic deviations from optimal behavior and suggest that they take such deviations into account when inferring preferences.

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Dec 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 4; 2019: 0; 2020: 1; 2021: 8; 2022: 8; 2023: 7; 2024: 11; 2025: 3; 2026: 3.

45 papers.

Sort:

By area:

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2024Large Language Models Assume People are More Rational than We Really arecited by 54
  3. 2024Beyond Preferences in AI Alignmentcited by 49
  4. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  5. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  6. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  7. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  8. 2021Human irrationality: both bad and good for reward inferencecited by 25
  9. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  10. 2023The Neuro-Symbolic Inverse Planning Engine (NIPE): Modeling Probabilistic Social Inferences from Linguistic Inputscited by 23
  11. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  12. 2021Cognitive science as a source of forward and inverse models of human decisions for robotics and controlcited by 22
33 more
  1. 2021Modeling the Mistakes of Boundedly Rational Agents Within a Bayesian Theory of Mindcited by 19
  2. 2021Choice Set Misspecification in Reward Inferencecited by 13
  3. 2023Active Reward Learning from Multiple Teacherscited by 13
  4. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  5. 2021Combining Reward Information from Multiple Sourcescited by 7
  6. 2022Why we need biased AI -- How including cognitive and ethical machine biases can enhance AI systemscited by 7
  7. 2020POPO: Pessimistic Offline Policy Optimizationcited by 5
  8. 2022Best-Response Bayesian Reinforcement Learning with Bayes-adaptive POMDPs for Centaurscited by 5
  9. 2023Concept Alignment as a Prerequisite for Value Alignmentcited by 5
  10. 2023(Ir)rationality in AI: State of the Art, Research Challenges and Open Questionscited by 5
  11. 2018Reinforcement Learning and Inverse Reinforcement Learning with System 1 and System 2cited by 4
  12. 2022Zero-Shot Assistance in Sequential Decision Problemscited by 4
  13. 2024Walking the Values in Bayesian Inverse Reinforcement Learningcited by 4
  14. 2023Explaining by Imitating: Understanding Decisions by Interpretable Policy Learningcited by 3
  15. 2021Multi-Agent Inverse Reinforcement Learning: Suboptimal Demonstrations and Alternative Solution Conceptscited by 2
  16. 2022Limited or Biased: Modeling Sub-Rational Human Investors in Financial Marketscited by 2
  17. 2022Modeling Mobile Health Users as Reinforcement Learning Agentscited by 2
  18. 2023Discovering User Types: Mapping User Traits by Task-Specific Behaviors in Reinforcement Learningcited by 2
  19. 2024Goal Recognition Design for General Behavioral Agents using Machine Learningcited by 2
  20. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  21. 2021A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problemscited by 1
  22. 2021Modeling human intention inference in continuous 3D domains by inverse planning and body kinematicscited by 1
  23. 2024Intuitions of Compromise: Utilitarianism vs. Contractualismcited by 1
  24. 2024Partial Identifiability in Inverse Reinforcement Learning For Agents With Non-Exponential Discountingcited by 1
  25. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  26. 2026Attention Limited Reward Learningcited by 1
  27. 2018Exploring Hierarchy-Aware Inverse Reinforcement Learningcited by 0
  28. 2018A Roadmap for Robust End-to-End Alignmentcited by 0
  29. 2024On the Utility of Accounting for Human Beliefs about AI Intention in Human-AI Collaborationcited by 0
  30. 2024Choice Between Partial Trajectories: Disentangling Goals from Beliefscited by 0
  31. 2025Supervised Reward Inferencecited by 0
  32. 2026Learning the Preferences of a Learning Agentcited by 0
  33. 2026Propose to Learn, Learn to Propose: Evaluability-Aware Assistance under Bounded Rationalitycited by 0