mlsanityData to 7 Oct 2026

SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning

arXiv 2203.10050 ↗First posted 18 Mar 2022cs.LG cs.AIcited by 96

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (RL) has shown potential for teaching agents to perform the target tasks without a costly, pre-defined reward function by learning the reward with a supervisor's preference between the two agent behaviors. However, preference-based learning often requires a large amount of human feedback, making it difficult to apply this approach to various applications. This data-efficiency problem, on the other hand, has been typically addressed by using unlabeled samples or data augmentation techniques in the context of supervised learning. Motivated by the recent success of these approaches, we present SURF, a semi-supervised reward learning framework that utilizes a large amount of unlabeled samples with data augmentation. In order to leverage unlabeled samples for reward learning, we infer pseudo-labels of the unlabeled samples based on the confidence of the preference predictor. To further improve the label-efficiency of reward learning, we introduce a new data augmentation that temporally crops consecutive subsequences from the original behaviors. Our experiments demonstrate that our approach significantly improves the feedback-efficiency of the state-of-the-art preference-based method on a variety of locomotion and robotic manipulation tasks.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 57 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

36 of 57 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  2. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
19 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  2. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  3. 2018DeepMind Control Suitecited by 935
  4. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  8. 2018Learning Dexterous In-Hand Manipulationcited by 643
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  12. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  13. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  14. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  15. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  16. 2017Inverse Reward Designcited by 246
  17. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  18. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
  19. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 22; 2024: 26; 2025: 25; 2026: 16.

96 papers.

Sort:

By area:

  1. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
  2. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  3. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  4. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  5. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  6. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  7. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  8. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  9. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  10. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  11. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  12. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
38 more
  1. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  2. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  3. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  4. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  5. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20
  6. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 19
  8. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  9. 2023Rating-based Reinforcement Learningcited by 14
  10. 2023SemiReward: A General Reward Model for Semi-supervised Learningcited by 14
  11. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  12. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  13. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
  14. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  15. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  16. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  17. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferencescited by 8
  18. 2024Semi-Supervised Reward Modeling via Iterative Self-Trainingcited by 8
  19. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 8
  20. 2022Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferencescited by 7
  21. 2024ICPL: Few-shot In-context Preference Learning via LLMscited by 7
  22. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  23. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  24. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  25. 2024Advances in Preference-based Reinforcement Learning: A Reviewcited by 6
  26. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 6
  27. 2022Efficient Meta Reinforcement Learning for Preference-based Fast Adaptationcited by 4
  28. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learningcited by 4
  29. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  30. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learningcited by 3
  31. 2023Data Driven Reward Initialization for Preference based Reinforcement Learningcited by 3
  32. 2024Preference Elicitation for Offline Reinforcement Learningcited by 3
  33. 2024From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledgecited by 3
  34. 2025Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignmentcited by 3
  35. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
  36. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  37. 2025Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedbackcited by 3
  38. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  2. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  3. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  4. 2026Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers
  5. 2020Semi-supervised reward learning for offline reinforcement learning
  6. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  7. 2025Diffusion Classifier-Driven Reward for Offline Preference-based Reinforcement Learning
  8. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoning