mlsanityData to 14 Oct 2026

Learning the Preferences of a Learning Agent

arXiv 2605.09217 ↗First posted 9 May 2026cs.AI cs.LG cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

For AI systems to be useful to humans, they must understand and act in accordance with our values and preferences. Since specifying preferences is a hard task, inverse reinforcement learning (IRL) aims to develop methods that allow for inferring preferences from observed behavior. However, IRL assumes the human to be approximately optimal. This is a big limitation in cases where the human themselves may be learning to act optimally in an environment. In this paper, we formalize the problem of learning the preferences of a learning agent: a predictor observes a learner acting online and tries to infer the underlying reward function being (initially suboptimally) optimized by the learner. We model the learner as either being no-regret, or as converging to an optimal Boltzmann policy over time. In each of these settings, we establish theoretical guarantees for various preference learning algorithms, or otherwise show that such guarantees are impossible.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

15 of 29 references.

References · 15

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  6. 2020Quantifying Differences in Reward Functionscited by 63
  7. 2021Identifiability in inverse reinforcement learningcited by 55
  8. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  9. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  10. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behaviorcited by 40
  11. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  12. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
3 more
  1. 2019The Assistive Multi-Armed Banditcited by 19
  2. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  3. 2021Uncertain Decisions Facilitate Better Preference Learningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2019Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  3. 2025Learning Pareto-Optimal Rewards from Noisy Preferences: A Framework for Multi-Objective Inverse Reinforcement Learning
  4. 2020Inverse Reinforcement Learning from a Gradient-based Learner
  5. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  6. 2021Prior Preference Learning from Experts:Designing a Reward with Active Inference
  7. 2022A Ranking Game for Imitation Learning
  8. 2026A Lecture Note on Offline RL and IRL, Part II: Foundations of Inverse Reinforcement Learning and Dynamic Discrete Choice Models