mlsanityData to 7 Oct 2026

Few-Shot Preference Learning for Human-in-the-Loop RL

arXiv 2212.03363 ↗First posted 6 Dec 2022cs.RO cs.AI cs.LGcited by 85

Not on the map. Why?

Cited by 45 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While reinforcement learning (RL) has become a more popular approach for robotics, designing sufficiently informative reward functions for complex tasks has proven to be extremely difficult due their inability to capture human intent and policy exploitation. Preference based RL algorithms seek to overcome these challenges by directly learning reward functions from human feedback. Unfortunately, prior work either requires an unreasonable number of queries implausible for any human to answer or overly restricts the class of reward functions to guarantee the elicitation of the most informative queries, resulting in models that are insufficiently expressive for realistic robotics tasks. Contrary to most works that focus on query selection to minimize the amount of data required for learning reward functions, we take an opposite approach: expanding the pool of available data by viewing human-in-the-loop RL through the more flexible lens of multi-task learning. Motivated by the success of meta-learning, we pre-train preference models on prior task data and quickly adapt them for new tasks using only a handful of queries. Empirically, we reduce the amount of online feedback needed to train manipulation policies in Meta-World by 20×, and demonstrate the effectiveness of our method on a real Franka Panda Robot. Moreover, this reduction in query-complexity allows us to train robot policies from actual human users. Videos of our results and code can be found at https://sites.google.com/view/few-shot-preference-rl/home.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Builds on 42 of its 66 references that are in the map. First posted on arXiv 6 Dec 2022.

Builds on

42 of 66 references.

References · 42

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  8. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
  9. 2016Learning to reinforcement learncited by 369
  10. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  12. 2018How to train your MAMLcited by 306
30 more
  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  3. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  4. 2017Inverse Reward Designcited by 246
  5. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  6. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  7. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  8. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  9. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  10. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  11. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
  12. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  13. 2018Practical Obstacles to Deploying Active Learningcited by 68
  14. 2021Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answeringcited by 62
  15. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 60
  16. 2021Learning Multimodal Rewards from Rankingscited by 41
  17. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  18. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  19. 2018Guiding Policies with Language via Meta-Learningcited by 34
  20. 2020When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humanscited by 34
  21. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedbackcited by 34
  22. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  23. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  24. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  25. 2020Avoiding Side Effects in Complex Environmentscited by 24
  26. 2021Learning Human Objectives from Sequences of Physical Correctionscited by 21
  27. 2021Learning Latent Actions to Control Assistive Robotscited by 17
  28. 2016A First Look at Ad-block Detection: A New Arms Race on the Webcited by 15
  29. 2019The Green Choice: Learning and Influencing Human Decisions on Shared Roadscited by 14
  30. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 23; 2025: 32; 2026: 14.

85 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  3. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  4. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  5. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  6. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  7. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  8. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  9. 2024A Dense Reward View on Aligning Text-to-Image Diffusion with Preferencecited by 55
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  12. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
38 more
  1. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  2. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  3. 2025Generating Robot Constitutions & Benchmarks for Semantic Safetycited by 28
  4. 2023Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferencescited by 25
  5. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  6. 2024Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through f-divergence Minimizationcited by 15
  7. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  8. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  9. 2025OvercookedV2: Rethinking Overcooked for Zero-Shot Coordinationcited by 14
  10. 2023Active Reward Learning from Online Preferencescited by 12
  11. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  12. 2024Offline Imitation Learning Through Graph Search and Retrievalcited by 12
  13. 2025Segmenting Text and Learning Their Rewards for Improved RLHF in Language Modelcited by 12
  14. 2024Learning with Language-Guided State Abstractionscited by 11
  15. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  16. 2023PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulationcited by 9
  17. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  18. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  19. 2023Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrationscited by 8
  20. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  21. 2023Learning Human-like Representations to Enable Learning Human Valuescited by 7
  22. 2025Preference-based Multi-Objective Reinforcement Learningcited by 7
  23. 2024LORD: Large Models based Opposite Reward Design for Autonomous Drivingcited by 6
  24. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  25. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 6
  26. 2024It Takes Two: On the Seamlessness between Reward and Policy Model in RLHFcited by 5
  27. 2025Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarmcited by 5
  28. 2024Revision Matters: Generative Design Guided by Revision Editscited by 4
  29. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  30. 2025Influencing Humans to Conform to Preference Models for RLHFcited by 4
  31. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  32. 2025FLoRA: Sample-Efficient Preference-based RL via Low-Rank Style Adaptation of Reward Functionscited by 3
  33. 2025MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrationscited by 3
  34. 2025Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agentscited by 3
  35. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  36. 2024Leveraging Human Revisions for Improving Text-to-Layout Modelscited by 2
  37. 2024Multi-Type Preference Learning: Empowering Preference-Based Reinforcement Learning with Equal Preferencescited by 2
  38. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2

50 most cited of 85 All 85 (JSON)