mlsanityData to 7 Oct 2026

RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

arXiv 2308.04332 ↗First posted 8 Aug 2023cs.LG cs.HCcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

To use reinforcement learning from human feedback (RLHF) in practical applications, it is crucial to learn reward models from diverse sources of human feedback and to consider human factors involved in providing feedback of different types. However, the systematic study of learning from diverse types of feedback is held back by limited standardized tooling available to researchers. To bridge this gap, we propose RLHF-Blender, a configurable, interactive interface for learning from human feedback. RLHF-Blender provides a modular experimentation framework and implementation that enables researchers to systematically investigate the properties and qualities of human feedback for reward learning. The system facilitates the exploration of various feedback types, including demonstrations, rankings, comparisons, and natural language instructions, as well as studies considering the impact of human factors on their effectiveness. We discuss a set of concrete research opportunities enabled by RLHF-Blender. More information is available at https://rlhfblender.info/.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 38 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

20 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types23 Aug
  2. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  3. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  4. 2016The Off-Switch Gamecited by 97
  5. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  6. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  7. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  8. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  9. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  10. 2022imitation: Clean Imitation Learning Implementationscited by 34
  11. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  12. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
5 more
  1. 2022Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interactioncited by 18
  2. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  3. 2018Reward Learning from Narrated Demonstrationscited by 13
  4. 2022Linguistic communication as (inverse) reward designcited by 6
  5. 2021Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gymcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  3. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  4. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11
  5. 2025Reward Learning from Multiple Feedback Typescited by 9
  6. 2024Deconstructing Human-AI Collaboration: Agency, Interaction, and Adaptationcited by 8
  7. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  8. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  9. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decompositioncited by 3
  10. 2025Improving User Interface Generation Models from Designer Feedbackcited by 3
  11. 2025Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemmacited by 2
  12. 2024Tell me why: Training preferences-based RL with human preferences and step-level explanationscited by 1
3 more
  1. 2024On the Effect of Robot Errors on Human Teaching Dynamicscited by 0
  2. 2026Building Intelligent User Interfaces for Human-AI Alignmentcited by 0
  3. 2026MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey of Reinforcement Learning from Human Feedback
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  5. 2023The History and Risks of Reinforcement Learning and Human Feedback
  6. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  7. 2025Reinforcement Learning from Human Feedback
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective