# Primitive Skill-based Robot Learning from Human Evaluative Feedback

arXiv: https://arxiv.org/abs/2307.15801 · HTML: https://mlsanity.com/paper/2307.15801/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 24 of its 45 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 24 of 24)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning](https://mlsanity.com/paper/1802.08705/) (2018)
- [VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priors](https://mlsanity.com/paper/2210.11339/) (2022)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs](https://mlsanity.com/paper/2012.07277/) (2020)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Learning Neuro-Symbolic Relational Transition Models for Bilevel Planning](https://mlsanity.com/paper/2105.14074/) (2021)
- [Leveraging Human Guidance for Deep Reinforcement Learning Tasks](https://mlsanity.com/paper/1909.09906/) (2019)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [Deep Affordance Foresight: Planning Through What Can Be Done in the Future](https://mlsanity.com/paper/2011.08424/) (2020)
- [LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulation](https://mlsanity.com/paper/2210.12631/) (2022)
- [Generalizable Task Planning through Representation Pretraining](https://mlsanity.com/paper/2205.07993/) (2022)
- [STAP: Sequencing Task-Agnostic Policies](https://mlsanity.com/paper/2210.12250/) (2022)
- [APPLE: Adaptive Planner Parameter Learning from Evaluative Feedback](https://mlsanity.com/paper/2108.09801/) (2021)
- [Accelerated Robot Learning via Human Brain Signals](https://mlsanity.com/paper/1910.00682/) (2019)
- [Interactively shaping robot behaviour with unlabeled human instructions](https://mlsanity.com/paper/1902.01670/) (2019)

## Built on by (most cited first, 15 of 15)

- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [Principled Preferential Bayesian Optimization](https://mlsanity.com/paper/2402.05367/) (2024)
- [NOIR: Neural Signal Operated Intelligent Robots for Everyday Activities](https://mlsanity.com/paper/2311.01454/) (2023)
- [VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training](https://mlsanity.com/paper/2403.08109/) (2024)
- [Learning Diffusion Policy from Primitive Skills for Robot Manipulation](https://mlsanity.com/paper/2601.01948/) (2026)
- [Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Space](https://mlsanity.com/paper/2505.17389/) (2025)
- [Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning](https://mlsanity.com/paper/2504.08772/) (2025)
- [HAPI: A Model for Learning Robot Facial Expressions from Human Preferences](https://mlsanity.com/paper/2503.17046/) (2025)
- [Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.23605/) (2026)
- [CasIL: Cognizing and Imitating Skills via a Dual Cognition-Action Architecture](https://mlsanity.com/paper/2309.16299/) (2023)
- [NOIR 2.0: Neural Signal Operated Intelligent Robots for Everyday Activities](https://mlsanity.com/paper/2511.20848/) (2025)
- [Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMs](https://mlsanity.com/paper/2606.18747/) (2026)
- [PIVOT: Pivot-Aware On Policy Self Distillation for Multi-Turn VLM Agents](https://mlsanity.com/paper/2609.35303/) (2026)
- [Beyond Prediction: Steering VLM Agents with Retrospective World Modeling](https://mlsanity.com/paper/2609.39101/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) algorithms face significant challenges when dealing with long-horizon robot manipulation tasks in real-world environments due to sample inefficiency and safety issues. To overcome these challenges, we propose a novel framework, SEED, which leverages two approaches: reinforcement learning from human feedback (RLHF) and primitive skill-based reinforcement learning. Both approaches are particularly effective in addressing sparse reward issues and the complexities involved in long-horizon tasks. By combining them, SEED reduces the human effort required in RLHF and increases safety in training robot manipulation with RL in real-world settings. Additionally, parameterized skills provide a clear view of the agent's high-level intentions, allowing humans to evaluate skill choices before they are executed. This feature makes the training process even safer and more efficient. To evaluate the performance of SEED, we conducted extensive experiments on five manipulation tasks with varying levels of complexity. Our results show that SEED significantly outperforms state-of-the-art RL algorithms in sample efficiency and safety. In addition, SEED also exhibits a substantial reduction of human effort compared to other RLHF methods. Further details and video results can be found at https://seediros23.github.io/.