mlsanityData to 14 Oct 2026

LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency

arXiv 2412.21001 ↗First posted 30 Dec 2024cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Offline preference-based reinforcement learning (PbRL) provides an effective way to overcome the challenges of designing reward and the high costs of online interaction. However, since labeling preference needs real-time human feedback, acquiring sufficient preference labels is challenging. To solve this, this paper proposes a offLine prEference-bAsed RL with high Sample Efficiency (LEASE) algorithm, where a learned transition model is leveraged to generate unlabeled preference data. Considering the pretrained reward model may generate incorrect labels for unlabeled data, we design an uncertainty-aware mechanism to ensure the performance of reward model, where only high confidence and low variance data are selected. Moreover, we provide the generalization bound of reward model to analyze the factors influencing reward accuracy, and demonstrate that the policy learned by LEASE has theoretical improvement guarantee. The developed theory is based on state-action pair, which can be easily combined with other offline algorithms. The experimental results show that LEASE can achieve comparable performance to baseline under fewer preference data without online interaction.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 64 references that are in the map. First posted on arXiv 30 Dec 2024.

Builds on

38 of 64 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Hindsight PRIORs for Reward Learning from Human Preferences12 Apr
  2. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback4 Feb
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  5. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  6. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  7. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  4. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
18 more
  1. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  2. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  3. 2017Inverse Reward Designcited by 249
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 238
  5. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  6. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  7. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
  8. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  9. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  10. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  11. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  12. 2022ProMix: Combating Label Noise via Maximizing Clean Sample Utilitycited by 48
  13. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  14. 2023Class-Distribution-Aware Pseudo Labeling for Semi-Supervised Multi-Label Learningcited by 16
  15. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
  16. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  17. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  18. 2020Towards Expedited Impedance Tuning of a Robotic Prosthesis for Personalized Gait Assistance by Reinforcement Learning Controlcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learning
  2. 2024Binary Reward Labeling: Bridging Offline Preference and Reward-Based Reinforcement Learning
  3. 2024Preference Elicitation for Offline Reinforcement Learning
  4. 2024Online Policy Learning from Offline Preferences
  5. 2023Direct Preference-based Policy Optimization without Reward Modeling
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning
  7. 2025Adversarial Policy Optimization for Offline Preference-based Reinforcement Learning
  8. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learning