mlsanityData to 7 Oct 2026

Invariance in Policy Optimisation and Partial Identifiability in Reward Learning

arXiv 2203.07475 ↗First posted 14 Mar 2022cs.LG cs.AI stat.MLcited by 46

Not on the map. Why?

Abstract (arXiv)

It is often very challenging to manually design reward functions for complex, real-world tasks. To solve this, one can instead use reward learning to infer a reward function from data. However, there are often multiple reward functions that fit the data equally well, even in the infinite-data limit. This means that the reward function is only partially identifiable. In this work, we formally characterise the partial identifiability of the reward function given several popular reward learning data sources, including expert demonstrations and trajectory comparisons. We also analyse the impact of this partial identifiability for several downstream tasks, such as policy optimisation. We unify our results in a framework for comparing data sources and downstream tasks by their invariances, with implications for the design and selection of data sources for reward learning.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 21 of its 66 references that are in the map. First posted on arXiv 14 Mar 2022.

Builds on

21 of 66 references.

References · 21

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  6. 2020Artificial Intelligence, Values and Alignmentcited by 443
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  8. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  9. 2019Challenges of Real-World Reinforcement Learningcited by 282
  10. 2020Open Problems in Cooperative AIcited by 167
  11. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  12. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
9 more
  1. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  2. 2021What Matters for Adversarial Imitation Learning?cited by 70
  3. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  4. 2021Identifiability in inverse reinforcement learningcited by 55
  5. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  6. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 44
  7. 2017Repeated Inverse Reinforcement Learningcited by 29
  8. 2018Reward Learning from Narrated Demonstrationscited by 13
  9. 2021Combining Reward Information from Multiple Sourcescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 10; 2024: 14; 2025: 9; 2026: 5.

46 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  3. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 94
  4. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  5. 2022Models of human preference for learning reward functionscited by 54
  6. 2024Beyond Preferences in AI Alignmentcited by 46
  7. 2023Goodhart's Law in Reinforcement Learningcited by 37
  8. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  9. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  10. 2023On The Fragility of Learned Reward Functionscited by 21
  11. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  12. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
34 more
  1. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  2. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
  3. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 12
  4. 2022Preprocessing Reward Functions for Interpretabilitycited by 7
  5. 2023Automatic Pair Construction for Contrastive Post-trainingcited by 7
  6. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learningcited by 6
  7. 2022Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemblecited by 5
  8. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  9. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learningcited by 5
  10. 2025Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hackingcited by 5
  11. 2022Calculus on MDPs: Potential Shaping as a Gradientcited by 4
  12. 2023FoMo Rewards: Can we cast foundation models as reward functions?cited by 4
  13. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4
  14. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4
  15. 2025Inverse Reinforcement Learning with Just Classification and a Few Regressionscited by 4
  16. 2024Enhancing Inverse Reinforcement Learning through Encoding Dynamic Information in Reward Shapingcited by 3
  17. 2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
  18. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
  19. 2025The Limits of Predicting Agents from Behaviourcited by 3
  20. 2026Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillationcited by 3
  21. 2023Eliciting Risk Aversion with Inverse Reinforcement Learning via Interactive Questioningcited by 2
  22. 2024A Generalized Acquisition Function for Preference-based Reward Learningcited by 1
  23. 2025Efficient Reward Identification In Max Entropy Reinforcement Learning with Sparsity and Rank Priorscited by 1
  24. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroidscited by 1
  25. 2025Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RLcited by 1
  26. 2022Bounded Robustness in Reinforcement Learning via Lexicographic Objectivescited by 0
  27. 2023PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Rewardcited by 0
  28. 2024Non-maximizing policies that fulfill multi-criterion aspirations in expectationcited by 0
  29. 2025Robustness in the Face of Partial Identifiability in Reward Learningcited by 0
  30. 2025Statistical analysis of Inverse Entropy-regularized Reinforcement Learningcited by 0
  31. 2026Learning the Preferences of a Learning Agentcited by 0
  32. 2026Understanding Goal Generalisation in Sequential Reinforcement Learningcited by 0
  33. 2026Q-based Variational Inverse Reinforcement Learningcited by 0
  34. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimizationcited by 0