mlsanityData to 7 Oct 2026

Diversity from Human Feedback

arXiv 2310.06648 ↗First posted 10 Oct 2023cs.LG cs.AI cs.NEcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Diversity plays a significant role in many problems, such as ensemble learning, reinforcement learning, and combinatorial optimization. How to define the diversity measure is a longstanding problem. Many methods rely on expert experience to define a proper behavior space and then obtain the diversity measure, which is, however, challenging in many scenarios. In this paper, we propose the problem of learning a behavior space from human feedback and present a general method called Diversity from Human Feedback (DivHF) to solve it. DivHF learns a behavior descriptor consistent with human preference by querying human feedback. The learned behavior descriptor can be combined with any distance measure to define a diversity measure. We demonstrate the effectiveness of DivHF by integrating it with the Quality-Diversity optimization algorithm MAP-Elites and conducting experiments on the QDax suite. The results show that the behavior learned by DivHF is much more consistent with human requirements than the one learned by direct data-driven approaches without human feedback, and makes the final solutions more diverse under human preference. Our contributions include formulating the problem, proposing the DivHF method, and demonstrating its effectiveness through experiments.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 84 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

46 of 84 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  2. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  3. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 42

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2015Illuminating search spaces by mapping elitescited by 406
  8. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  10. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  11. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  12. 2014Robots that can adapt like animalscited by 203
30 more
  1. 2020Prioritized Level Replaycited by 190
  2. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  3. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  4. 2017Quality and Diversity Optimization: A Unifying Modular Frameworkcited by 111
  5. 2021Replay-Guided Adversarial Environment Designcited by 108
  6. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  7. 2019Covariance Matrix Adaptation for the Rapid Illumination of Behavior Spacecited by 78
  8. 2021Differentiable Quality Diversitycited by 71
  9. 2020Quality-Diversity Optimization: a novel branch of stochastic optimizationcited by 59
  10. 2019Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptorscited by 57
  11. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 53
  12. 2020Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Networkcited by 51
  13. 2015Transition-Based Dependency Parsing with Stack Long Short-Term Memorycited by 45
  14. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  15. 2020Scaling MAP-Elites to Deep Neuroevolutioncited by 38
  16. 2021Unsupervised Behaviour Discovery with Quality-Diversity Optimisationcited by 30
  17. 2022Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discoverycited by 23
  18. 2019Empowering Quality Diversity in Dungeon Design with Interactive Constrained MAP-Elitescited by 18
  19. 2023Multi-Robot Coordination and Layout Design for Automated Warehousingcited by 17
  20. 2023QDax: A Library for Quality-Diversity and Population-based Algorithms with Hardware Accelerationcited by 16
  21. 2023Policy Space Diversity for Non-Transitive Gamescited by 15
  22. 2022Accelerated Quality-Diversity through Massive Parallelismcited by 14
  23. 2023Reducing Idleness in Financial Cloud Services via Multi-objective Evolutionary Reinforcement Learning based Load Balancercited by 13
  24. 2020Interactive Constrained MAP-Elites: Analysis and Evaluation of the Expressiveness of the Feature Dimensionscited by 11
  25. 2021CRD-CGAN: Category-Consistent and Relativistic Constraints for Diverse Text-to-Image Generationcited by 10
  26. 2022DNS: Determinantal Point Process Based Neural Network Sampler for Ensemble Reinforcement Learningcited by 8
  27. 2022Relevance-guided Unsupervised Discovery of Abilities with Quality-Diversity Algorithmscited by 8
  28. 2021Analysis of Evolutionary Diversity Optimisation for Permutation Problemscited by 7
  29. 2023Controlled Diversity with Preference : Towards Learning a Diverse Set of Desired Skillscited by 6
  30. 2022Computing High-Quality Solutions for the Patient Admission Scheduling Problem using Evolutionary Diversity Optimisationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agentscited by 294
  2. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
  3. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  4. 2024Quality-Diversity with Limited Resourcescited by 1
  5. 2026Data-dependent Exploration for Online Reinforcement Learning from Human Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization
  2. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  4. 2025Algorithmic Prompt Generation for Diverse Human-like Teaming and Communication with Large Language Models
  5. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
  6. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model
  7. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning