mlsanityData to 7 Oct 2026

Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignment

arXiv 2501.17431 ↗First posted 29 Jan 2025cs.LG cs.ROcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Unsupervised skill discovery in Reinforcement Learning aims to mimic humans' ability to autonomously discover diverse behaviors. However, existing methods are often unconstrained, making it difficult to find useful skills, especially in complex environments, where discovered skills are frequently unsafe or impractical. We address this issue by proposing Human-aligned Skill Discovery (HaSD), a framework that incorporates human feedback to discover safer, more aligned skills. HaSD simultaneously optimises skill diversity and alignment with human values. This approach ensures that alignment is maintained throughout the skill discovery process, eliminating the inefficiencies associated with exploring unaligned skills. We demonstrate its effectiveness in both 2D navigation and SafetyGymnasium environments, showing that HaSD discovers diverse, human-aligned skills that are safe and useful for downstream tasks. Finally, we extend HaSD by learning a range of configurable skills with varying degrees of diversity alignment trade-offs that could be useful in practical scenarios.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 32 references that are in the map. First posted on arXiv 29 Jan 2025.

Builds on

21 of 32 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  2. 2021Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback11 Aug
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 17

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  4. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  5. 2016Variational Intrinsic Controlcited by 262
  6. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  7. 2018Variational Option Discovery Algorithmscited by 121
  8. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  9. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  10. 2017Trial without Error: Towards Safe Reinforcement Learning via Human Interventioncited by 102
  11. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  12. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
5 more
  1. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  2. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  3. 2021Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Taskscited by 13
  4. 2023Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skillscited by 6
  5. 2021Learning Task Agnostic Skills with Data-driven Guidancecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  2. 2025Controllable Complementarity: Subjective Preferences in Human-AI Collaborationcited by 0
  3. 2026Leveraging Human Feedback for Semantically-Relevant Skill Discoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DSD: Learning Diverse and Reusable Motor Skills via Diffusion Skill Discovery
  2. 2023Offline Diversity Maximization Under Imitation Constraints
  3. 2026Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents
  4. 2024Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment
  5. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
  6. 2025Curiosity-Driven Reinforcement Learning from Human Feedback
  7. 2026Diverse Skill Discovery for Quadruped Robots via Unsupervised Learning
  8. 2026Search2Skill: Skill Distillation Beyond Knowledge Boundaries Via Rubric-Based Reinforcement Learning