mlsanityData to 7 Oct 2026

PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training

arXiv 2106.05091 ↗First posted 9 Jun 2021cs.LG cs.AIcited by 298

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Conveying complex objectives to reinforcement learning (RL) agents can often be difficult, involving meticulous design of reward functions that are sufficiently informative yet easy enough to provide. Human-in-the-loop RL methods allow practitioners to instead interactively teach agents through tailored feedback; however, such approaches have been challenging to scale since human feedback is very expensive. In this work, we aim to make this process more sample- and feedback-efficient. We present an off-policy, interactive RL algorithm that capitalizes on the strengths of both feedback and off-policy learning. Specifically, we learn a reward model by actively querying a teacher's preferences between two clips of behavior and use it to train an agent. To enable off-policy learning, we relabel all the agent's past experience when its reward model changes. We additionally show that pre-training our agents with unsupervised exploration substantially increases the mileage of its queries. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods, including a variety of locomotion and robotic manipulation skills. We also show that our method is able to utilize real-time human feedback to effectively prevent reward exploitation and learn new behaviors that are difficult to specify with standard reward functions.

Key facts. Cited by 298 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 47 of its 82 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

47 of 82 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020Weak Human Preference Supervision For Deep Reinforcement Learning25 Jul
  2. 2020Active Preference-Based Gaussian Process Regression for Reward Learning6 May
  3. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  4. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  5. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2016Concrete Problems in AI Safetycited by 1,582
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018Exploration by Random Network Distillationcited by 966
  10. 2018DeepMind Control Suitecited by 935
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
29 more
  1. 2019Solving Rubik's Cube with a Robot Handcited by 699
  2. 2018Learning Dexterous In-Hand Manipulationcited by 643
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  5. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  7. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  8. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  9. 2017Count-Based Exploration with Neural Density Modelscited by 255
  10. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  11. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 229
  12. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  13. 2019Efficient Exploration via State Marginal Matchingcited by 186
  14. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  15. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  16. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  17. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  18. 2018Variational Option Discovery Algorithmscited by 121
  19. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  20. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  21. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  22. 2016Sim-to-Real Robot Learning from Pixels with Progressive Netscited by 89
  23. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  24. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  25. 2019Preferences Implicit in the State of the Worldcited by 33
  26. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  27. 2016Visual Closed-Loop Control for Pouring Liquidscited by 26
  28. 2016Collective Robot Reinforcement Learning with Distributed Asynchronous Guided Policy Searchcited by 25
  29. 2020Avoiding Side Effects in Complex Environmentscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 19; 2023: 72; 2024: 83; 2025: 72; 2026: 47.

298 papers.

Sort:

By area:

  1. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  4. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  5. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  6. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 246
  7. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  8. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  9. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  11. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  12. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
38 more
  1. 2023Vision-Language Models as Success Detectorscited by 138
  2. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  5. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  6. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  7. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  8. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  9. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  10. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  11. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  12. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimismcited by 69
  13. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  14. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  15. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  16. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  17. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  18. 2022Models of human preference for learning reward functionscited by 54
  19. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  20. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  21. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  22. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  23. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Modelscited by 41
  24. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 39
  25. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedbackcited by 38
  26. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  27. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  28. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  29. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safetycited by 37
  30. 2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
  31. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  32. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 35
  33. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  34. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  35. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  36. 2024Reward-Robust RLHF in LLMscited by 32
  37. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
  38. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignmentcited by 27

50 most cited of 298 All 298 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  2. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  3. 2026Reward-Conditioned Reinforcement Learning
  4. 2022Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback
  5. 2018Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning
  6. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
  7. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  8. 2021Interactive Learning from Activity Description