mlsanityData to 7 Oct 2026

Positive-Unlabeled Reward Learning

arXiv 1911.00459 ↗First posted 1 Nov 2019cs.LG stat.MLcited by 36

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning reward functions from data is a promising path towards achieving scalable Reinforcement Learning (RL) for robotics. However, a major challenge in training agents from learned reward models is that the agent can learn to exploit errors in the reward model to achieve high reward behaviors that do not correspond to the intended task. These reward delusions can lead to unintended and even dangerous behaviors. On the other hand, adversarial imitation learning frameworks tend to suffer the opposite problem, where the discriminator learns to trivially distinguish agent and expert behavior, resulting in reward models that produce low reward signal regardless of the input state. In this paper, we connect these two classes of reward learning methods to positive-unlabeled (PU) learning, and we show that by applying a large-scale PU learning algorithm to the reward learning problem, we can address both the reward under- and over-estimation problems simultaneously. Our approach drastically improves both GAIL and supervised reward learning, without any additional assumptions.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 52 references that are in the map. First posted on arXiv 1 Nov 2019.

Builds on

28 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  2. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2018DeepMind Control Suitecited by 935
  7. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2018Distributed Prioritized Experience Replaycited by 260
  11. 2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 217
  12. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
13 more
  1. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  2. 2017Third-Person Imitation Learningcited by 103
  3. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  4. 2016Class-prior Estimation for Learning from Positive and Unlabeled Datacited by 46
  5. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  6. 2016Theoretical Comparisons of Positive-Unlabeled Learning against Positive-Negative Learningcited by 41
  7. 2018Sample-Efficient Imitation Learning via Generative Adversarial Netscited by 36
  8. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  9. 2018Classification from Positive, Unlabeled and Biased Negative Datacited by 34
  10. 2016Loss factorization, weakly supervised learning and label noise robustnesscited by 24
  11. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  12. 2016Avoiding Wireheading with Value Reinforcement Learningcited by 15
  13. 2019Reinforced Imitation in Heterogeneous Action Spacecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 5; 2021: 7; 2022: 4; 2023: 8; 2024: 6; 2025: 4; 2026: 2.

36 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  3. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  4. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  5. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  6. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  7. 2021What Matters for Adversarial Imitation Learning?cited by 70
  8. 2023Feed Two Birds with One Scone: Exploiting Wild Data for Both Out-of-Distribution Generalization and Detectioncited by 57
  9. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 52
  10. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  11. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  12. 2024How Does Unlabeled Data Provably Help Out-of-Distribution Detection?cited by 36
24 more
  1. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  2. 2023On The Fragility of Learned Reward Functionscited by 21
  3. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  4. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  5. 2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
  6. 2022Positive-Unlabeled Learning with Adversarial Data Augmentation for Knowledge Graph Completioncited by 13
  7. 2021A Pragmatic Look at Deep Imitation Learningcited by 9
  8. 2024DrS: Learning Reusable Dense Rewards for Multi-Stage Taskscited by 9
  9. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrationscited by 7
  10. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  11. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  12. 2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
  13. 2020Combating False Negatives in Adversarial Imitation Learningcited by 4
  14. 2025Out-of-Distribution Detection using Synthetic Data Generationcited by 4
  15. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  16. 2021Back to Reality for Imitation Learningcited by 3
  17. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learningcited by 3
  18. 2023Contrastive Example-Based Controlcited by 3
  19. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  20. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
  21. 2025Medix: Out-of-Distribution Detection from Unlabeled Wild Data via Robust Gradient Statisticscited by 2
  22. 2024Skill-Enhanced Reinforcement Learning Acceleration from Heterogeneous Demonstrationscited by 1
  23. 2025Skill-based Safe Reinforcement Learning with Risk Planningcited by 0
  24. 2026Latent Wasserstein Adversarial Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemble
  2. 2024Learning Transparent Reward Models via Unsupervised Feature Selection
  3. 2024The Distributional Reward Critic Framework for Reinforcement Learning Under Perturbed Rewards
  4. 2023Accelerating Exploration with Unlabeled Prior Data
  5. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  6. 2023Improving Behavioural Cloning with Positive Unlabeled Learning
  7. 2025Learning a Pessimistic Reward Model in RLHF
  8. 2024PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement