# Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skills

arXiv: https://arxiv.org/abs/2303.04592 · HTML: https://mlsanity.com/paper/2303.04592/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Variational Option Discovery Algorithms](https://mlsanity.com/paper/1807.10299/) (2018)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications](https://mlsanity.com/paper/1807.06333/) (2018)
- [Empowerment -- an Introduction](https://mlsanity.com/paper/1310.1863/) (2013)
- [Skill Preferences: Learning to Extract and Execute Robotic Skills from Human Feedback](https://mlsanity.com/paper/2108.05382/) (2021)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [Unsupervised Skill-Discovery and Skill-Learning in Minecraft](https://mlsanity.com/paper/2107.08398/) (2021)
- [Learning Task Agnostic Skills with Data-driven Guidance](https://mlsanity.com/paper/2108.01869/) (2021)

## Built on by (most cited first, 6 of 6)

- [Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2405.00746/) (2024)
- [Diversity from Human Feedback](https://mlsanity.com/paper/2310.06648/) (2023)
- [Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignment](https://mlsanity.com/paper/2501.17431/) (2025)
- [COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space](https://mlsanity.com/paper/2606.00950/) (2026)
- [Offline Policy Learning via Skill-step Abstraction for Long-horizon Goal-Conditioned Tasks](https://mlsanity.com/paper/2408.11300/) (2024)
- [Leveraging Human Feedback for Semantically-Relevant Skill Discovery](https://mlsanity.com/paper/2604.24127/) (2026)

## Abstract (arXiv)

Autonomously learning diverse behaviors without an extrinsic reward signal has been a problem of interest in reinforcement learning. However, the nature of learning in such mechanisms is unconstrained, often resulting in the accumulation of several unusable, unsafe or misaligned skills. In order to avoid such issues and ensure the discovery of safe and human-aligned skills, it is necessary to incorporate humans into the unsupervised training process, which remains a largely unexplored research area. In this work, we propose Controlled Diversity with Preference (CDP), a novel, collaborative human-guided mechanism for an agent to learn a set of skills that is diverse as well as desirable. The key principle is to restrict the discovery of skills to those regions that are deemed to be desirable as per a preference model trained using human preference labels on trajectory pairs. We evaluate our approach on 2D navigation and Mujoco environments and demonstrate the ability to discover diverse, yet desirable skills.