mlsanityData to 7 Oct 2026

Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards

arXiv 2402.17975 ↗First posted 28 Feb 2024cs.AI cs.LGcited by 9

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (PbRL) aligns a robot behavior with human preferences via a reward function learned from binary feedback over agent behaviors. We show that dynamics-aware reward functions improve the sample efficiency of PbRL by an order of magnitude. In our experiments we iterate between: (1) learning a dynamics-aware state-action representation (zsa) via a self-supervised temporal consistency task, and (2) bootstrapping the preference-based reward function from (zsa), which results in faster policy learning and better final policy performance. For example, on quadruped-walk, walker-walk, and cheetah-run, with 50 preference labels we achieve the same performance as existing approaches with 500 preference labels, and we recover 83% and 66% of ground truth reward policy performance versus only 38% and 21%. The performance gains demonstrate the benefits of explicitly learning a dynamics-aware reward model. Repo: texttthttps://github.com/apple/ml-reed.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 60 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

40 of 60 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback11 Aug
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  6. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  7. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  8. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2016Concrete Problems in AI Safetycited by 1,582
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2018DeepMind Control Suitecited by 935
  12. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
19 more
  1. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  2. 2016Cooperative Inverse Reinforcement Learningcited by 298
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  5. 2017Inverse Reward Designcited by 246
  6. 2021Mastering Atari Games with Limited Datacited by 235
  7. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  8. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  9. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  10. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  13. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  14. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  15. 2020Deep Reinforcement and InfoMax Learningcited by 79
  16. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  17. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
  18. 2022A Ranking Game for Imitation Learningcited by 18
  19. 2023SIRL: Similarity-based Implicit Representation Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  2. 2025On The Sample Complexity Bounds In Bilevel Reinforcement Learningcited by 10
  3. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  4. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 3
  5. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  6. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hackingcited by 1
  7. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  8. 2026Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learningcited by 0
  9. 2026PrefPI: Preference-Guided Steering into Out-of-Distribution Behaviorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functions
  2. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
  3. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning
  4. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  5. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning
  6. 2023Direct Preference-based Policy Optimization without Reward Modeling
  7. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Robotics
  8. 2023Inverse Preference Learning: Preference-based RL without a Reward Function