mlsanityData to 7 Oct 2026

Supervised Reward Inference

arXiv 2502.18447 ↗First posted 25 Feb 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Existing approaches to reward inference typically assume that humans provide demonstrations according to specific behavior models. However, humans often indicate their goals through a wide range of behaviors, from actions that are suboptimal due to poor planning or execution to behaviors intended to communicate goals rather than achieve them. One existing solution for inferring rewards from such behavior unicodex2013 provided it is drawn from the same distribution at training and deployment unicodex2013 is to construct a dataset of behavior paired with known rewards, and to learn the mapping from behavior to rewards; however, prior methods in this family face notable limitations, such as restrictions to tabular settings. Given such a dataset, we propose instead that supervised learning offers a parsimonious yet powerful solution, which we term Supervised Reward Inference (SRI). Theoretically, we prove that SRI is asymptotically Bayes-optimal under standard assumptions. Empirically, SRI achieves near-ceiling performance on a prior benchmark for reward inference from suboptimal behavior, while on Meta-World robotics tasks, it infers rewards from even arbitrarily suboptimal demonstrations as accurately as those demonstrations allow. Finally, we demonstrate our framework's universality with straightforward generalizations to action- and goal-prediction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

43 of 79 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2017Deep Setscited by 1,394
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  12. 2016Cooperative Inverse Reinforcement Learningcited by 298
31 more
  1. 2017Inverse Reward Designcited by 246
  2. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  3. 2016Value Iteration Networkscited by 158
  4. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  5. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  6. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  7. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  8. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 68
  9. 2020Online Bayesian Goal Inference for Boundedly-Rational Planning Agentscited by 67
  10. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  11. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  12. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  13. 2017Occam's razor is insufficient to infer the preferences of irrational agentscited by 44
  14. 2018Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behaviorcited by 40
  15. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  16. 2018Learning a Prior over Intent via Meta-Inverse Reinforcement Learningcited by 36
  17. 2022imitation: Clean Imitation Learning Implementationscited by 34
  18. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  19. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 30
  20. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  21. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Typescited by 26
  22. 2021Human irrationality: both bad and good for reward inferencecited by 25
  23. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  24. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 17
  25. 2018An Efficient, Generalized Bellman Update For Cooperative Inverse Reinforcement Learningcited by 15
  26. 2023Active Reward Learning from Multiple Teacherscited by 13
  27. 2017Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methodscited by 12
  28. 2018Simplifying Reward Design through Divide-and-Conquercited by 12
  29. 2022Lifelong Inverse Reinforcement Learningcited by 9
  30. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  31. 2021Meta-Adversarial Inverse Reinforcement Learning for Decision-making Taskscited by 6

Built on by

Not cited yet.