mlsanityData to 7 Oct 2026

Leveraging Human Feedback for Semantically-Relevant Skill Discovery

arXiv 2604.24127 ↗First posted 27 Apr 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Unsupervised skill discovery in reinforcement learning aims to intrinsically motivate agents to discover diverse and useful behaviours. However, unconstrained approaches can produce unsafe, unethical, or misaligned behaviours. To mitigate these risks and improve the practical desireability of discovered skills, recent work grounds the discovery process by leveraging human preference feedback. However, preference-based approaches are feedback-inefficient and inherently ill-equipped to deal with skill spaces composed of a variety of different skills such as running, jumping, walking, etc. To overcome this limitation, we introduce semantic labelling, a novel and feedback-efficient approach that leverages human cognitive strengths to identify and label semantically meaningful behaviours. Based on semantic labelling, we propose Semantically Relevant Skill Discovery (SRSD), a novel human-in-the-loop approach that collects semantic labels from human feedback and learns a reward function to encourage skills to be more semantically diverse and relevant. Through our experiments in a 2D navigation environment and four locomotion environments, we demonstrate that SRSD can improve semantic diversity and discover relevant behaviours while scaling effectively to a large variety of behaviours.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 29 of its 41 references that are in the map. First posted on arXiv 27 Apr 2026.

Builds on

29 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignment29 Jan
  2. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 25

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018DeepMind Control Suitecited by 935
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  9. 2016Variational Intrinsic Controlcited by 262
  10. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  11. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  12. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
13 more
  1. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  2. 2018Variational Option Discovery Algorithmscited by 121
  3. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  4. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  5. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  6. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  7. 2023LOTUS: Continual Imitation Learning for Robot Manipulation Through Unsupervised Skill Discoverycited by 63
  8. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  9. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  10. 2016Learning Mixtures of Plackett-Luce Modelscited by 19
  11. 2024Language Guided Skill Discoverycited by 16
  12. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  13. 2023Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skillscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Semantic Bandits: In-Context Exploration-Exploitation is Biased by Semantic Priors
  2. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teachers
  3. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models
  4. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  5. 2025LAPP: Large Language Model Feedback for Preference-Driven Reinforcement Learning
  6. 2026DSD: Learning Diverse and Reusable Motor Skills via Diffusion Skill Discovery
  7. 2023Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning
  8. 2024REvolve: Reward Evolution with Large Language Models using Human Feedback