mlsanityData to 7 Oct 2026

Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback

arXiv 2108.05382 ↗First posted 11 Aug 2021cs.LG cs.ROcited by 34

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

A promising approach to solving challenging long-horizon tasks has been to extract behavior priors (skills) by fitting generative models to large offline datasets of demonstrations. However, such generative models inherit the biases of the underlying data and result in poor and unusable skills when trained on imperfect demonstration data. To better align skill extraction with human intent we present Skill Preferences (SkiP), an algorithm that learns a model over human preferences and uses it to extract human-aligned skills from offline data. After extracting human-preferred skills, SkiP also utilizes human feedback to solve down-stream tasks with RL. We show that SkiP enables a simulated kitchen robot to solve complex multi-step manipulation tasks and substantially outperforms prior leading RL algorithms with human preferences as well as leading skill extraction algorithms without human preferences.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 37 references that are in the map. First posted on arXiv 11 Aug 2021.

Builds on

28 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  2. 2020Weak Human Preference Supervision For Deep Reinforcement Learning25 Jul
  3. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  4. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  5. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2019Solving Rubik's Cube with a Robot Handcited by 699
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  11. 2017Inverse Reward Designcited by 246
  12. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
9 more
  1. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  2. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  3. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  4. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  5. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  6. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  7. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  8. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  9. 2020Avoiding Side Effects in Complex Environmentscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 7; 2024: 8; 2025: 3; 2026: 5.

34 papers.

Sort:

By area:

  1. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  2. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  3. 2022Models of human preference for learning reward functionscited by 54
  4. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  5. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  6. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  7. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  8. 2024Diffusion Model for Data-Driven Black-Box Optimizationcited by 24
  9. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  10. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 20
  11. 2023Learning Optimal Advantage from Preferences and Mistaking it for Rewardcited by 18
  12. 2022Transformers are Adaptable Task Plannerscited by 15
22 more
  1. 2023Provably Feedback-Efficient Reinforcement Learning via Active Reward Learningcited by 15
  2. 2024Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewardscited by 9
  3. 2024Efficient Preference-based Reinforcement Learning via Aligned Experience Estimationcited by 9
  4. 2024Large Language Model guided Deep Reinforcement Learning for Decision Making in Autonomous Drivingcited by 8
  5. 2022ASPiRe:Adaptive Skill Priors for Reinforcement Learningcited by 7
  6. 2023Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skillscited by 6
  7. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  8. 2025Influencing Humans to Conform to Preference Models for RLHFcited by 4
  9. 2022Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learningcited by 3
  10. 2022Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learningcited by 3
  11. 2025Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignmentcited by 3
  12. 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
  13. 2023Open Problems in Applied Deep Learningcited by 1
  14. 2024MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Interventioncited by 1
  15. 2025Preference Learning with Response Time: Robust Losses and Guaranteescited by 1
  16. 2023Sample Complexity of Preference-Based Nonparametric Off-Policy Evaluation with Deep Networkscited by 0
  17. 2023Socially Cognizant Robotics for a Technology Enhanced Societycited by 0
  18. 2026Enabling Option Learning in Sparse Rewards with Hindsight Experience Replaycited by 0
  19. 2026ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Taskscited by 0
  20. 2026Implicit Safety Alignment from Crowd Preferencescited by 0
  21. 2026Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMscited by 0
  22. 2026Rethinking the Implications of Human Feedback for Preference Learning in Human-Robot Collaborationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  2. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction
  3. 2026Freeform Preference Learning for Robotic Manipulation
  4. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preference
  5. 2023Offline Skill Generalization via Task and Motion Planning
  6. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  7. 2026Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation
  8. 2023XSkill: Cross Embodiment Skill Discovery