mlsanityData to 14 Oct 2026

Semi-supervised reward learning for offline reinforcement learning

arXiv 2012.06899 ↗First posted 12 Dec 2020cs.LG cs.AI cs.ROcited by 23

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

In offline reinforcement learning (RL) agents are trained using a logged dataset. It appears to be the most natural route to attack real-life applications because in domains such as healthcare and robotics interactions with the environment are either expensive or unethical. Training agents usually requires reward functions, but unfortunately, rewards are seldom available in practice and their engineering is challenging and laborious. To overcome this, we investigate reward learning under the constraint of minimizing human reward annotations. We consider two types of supervision: timestep annotations and demonstrations. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards. We further investigate the relationship between the quality of the reward model and the final policies. We notice, for example, that the reward models do not need to be perfect to result in useful policies.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 41 references that are in the map. First posted on arXiv 12 Dec 2020.

Builds on

25 of 41 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019Positive-Unlabeled Reward Learning1 Nov

Other references · 24

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  10. 2020Critic Regularized Regressioncited by 275
  11. 2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 219
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
12 more
  1. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  3. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 122
  4. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  5. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  6. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  7. 2017Learning human behaviors from motion capture by adversarial imitationcited by 77
  8. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  9. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 65
  10. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  11. 2015Weakly Supervised Object Localization with Multi-fold Multiple Instance Learningcited by 39
  12. 2016Perceptual Reward Functionscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 10; 2023: 3; 2024: 5; 2025: 2.

23 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  3. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 62
  4. 2021Offline Meta-Reinforcement Learning with Online Self-Supervisioncited by 53
  5. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  6. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  7. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  8. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  9. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 17
  10. 2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
  11. 2021Offline Inverse Reinforcement Learningcited by 14
  12. 2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
11 more
  1. 2022MLGOPerf: An ML Guided Inliner to Optimize Performancecited by 9
  2. 2022Reinforcement Guided Multi-Task Learning Framework for Low-Resource Stereotype Detectioncited by 7
  3. 2022Safe Deep RL in 3D Environments using Human Feedbackcited by 5
  4. 2024Multi-Fidelity Reinforcement Learning for Time-Optimal Quadrotor Re-planningcited by 4
  5. 2023Contrastive Example-Based Controlcited by 3
  6. 2022Semi-supervised Batch Learning From Logged Datacited by 2
  7. 2022Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learningcited by 1
  8. 2024Causal prompting model-based offline reinforcement learningcited by 1
  9. 2024Transductive Reward Inference on Graphcited by 0
  10. 2025Generalized Linear Markov Decision Processcited by 0
  11. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
  2. 2024Semi-Supervised Reward Modeling via Iterative Self-Training
  3. 2022Implicit Offline Reinforcement Learning via Supervised Learning
  4. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  6. 2019Reward-Conditioned Policies
  7. 2025Semi-pessimistic Reinforcement Learning
  8. 2024Offline Reinforcement Learning with Imputed Rewards