mlsanityData to 14 Oct 2026

Robustness in the Face of Partial Identifiability in Reward Learning

arXiv 2501.06376 ↗First posted 10 Jan 2025cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

In Reward Learning (ReL), we are given feedback on an unknown target reward, and the goal is to use this information to recover it in order to carry out some downstream application, e.g., planning. When the feedback is not informative enough, the target reward is only partially identifiable, i.e., there exists a set of rewards, called the feasible set, that are equally plausible candidates for the target reward. In these cases, the ReL algorithm might recover a reward function different from the target reward, possibly leading to a failure in the application. In this paper, we introduce a general ReL framework that permits to quantify the drop in "performance" suffered in the considered application because of identifiability issues. Building on this, we propose a robust approach to address the identifiability problem in a principled way, by maximizing the "performance" with respect to the worst-case reward in the feasible set. We then develop Rob-ReL, a ReL algorithm that applies this robust approach to the subset of ReL problems aimed at assessing a preference between two policies, and we provide theoretical guarantees on sample and iteration complexity for Rob-ReL. We conclude with a proof-of-concept experiment to illustrate the considered setting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 38 of its 64 references that are in the map. First posted on arXiv 10 Jan 2025.

Builds on

38 of 64 references.

References · 38

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2021Convex optimizationcited by 2,515
  3. 2016Generative Adversarial Imitation Learningcited by 1,597
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  8. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  10. 2016Cooperative Inverse Reinforcement Learningcited by 300
  11. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 298
  12. 2017Inverse Reward Designcited by 249
26 more
  1. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 247
  2. 2024Reinforcement learningcited by 247
  3. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  4. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  5. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
  6. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  7. 2021Identifiability in inverse reinforcement learningcited by 55
  8. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  9. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  10. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  11. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatchcited by 33
  12. 2021Online Apprenticeship Learningcited by 29
  13. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  14. 2022Active Exploration for Inverse Reinforcement Learningcited by 27
  15. 2023Towards Theoretical Understanding of Inverse Reinforcement Learningcited by 21
  16. 2016Towards Resolving Unidentifiability in Inverse Reinforcement Learningcited by 18
  17. 2024When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedbackcited by 17
  18. 2018Risk-Sensitive Generative Adversarial Imitation Learningcited by 16
  19. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learningcited by 16
  20. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  21. 2018Learning Safe Policies with Expert Guidancecited by 11
  22. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithmscited by 9
  23. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learningcited by 6
  24. 2024How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approachcited by 5
  25. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspectivecited by 4
  26. 2024Quantifying the Sensitivity of Inverse Reinforcement Learning to Misspecificationcited by 4

Built on by

Not cited yet.