mlsanityData to 7 Oct 2026

Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning

arXiv 2302.08738 ↗First posted 17 Feb 2023cs.RO cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference Based Reinforcement Learning has shown much promise for utilizing human binary feedback on queried trajectory pairs to recover the underlying reward model of the Human in the Loop (HiL). While works have attempted to better utilize the queries made to the human, in this work we make two observations about the unlabeled trajectories collected by the agent and propose two corresponding loss functions that ensure participation of unlabeled trajectories in the reward learning process, and structure the embedding space of the reward model such that it reflects the structure of state space with respect to action distances. We validate the proposed method on one locomotion domain and one robotic manipulation task and compare with the state-of-the-art baseline PEBBLE. We further present an ablation of the proposed loss components across both the domains and find that not only each of the loss components perform better than the baseline, but the synergic combination of the two has much better reward recovery and human feedback sample efficiency.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 28 references that are in the map. First posted on arXiv 17 Feb 2023.

Builds on

17 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Symbol Guided Hindsight Priors for Reward Learning from Human Preferences17 Oct
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 13

  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  2. 2017A Brief Survey of Deep Reinforcement Learningcited by 466
  3. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  4. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  5. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  6. 2021Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systemscited by 36
  7. 2020Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representationscited by 32
  8. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18
  9. 2021Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Graspingcited by 7
  10. 2022Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loopcited by 5
  11. 2022Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansioncited by 4
  12. 2019Making Smart Homes Smarter: Optimizing Energy Consumption with Human in the Loopcited by 3
1 more
  1. 2021Computing Policies That Account For The Effects Of Human Agent Uncertainty During Execution In Markov Decision Processescited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  2. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
  3. 2025SENIOR: Efficient Query Selection and Preference-Guided Exploration in Preference-based Reinforcement Learningcited by 1
  4. 2023Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A State Augmentation based approach to Reinforcement Learning from Human Preferences
  2. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
  3. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  4. 2020Weak Human Preference Supervision For Deep Reinforcement Learning
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Function
  6. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  7. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
  8. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedback