# RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

arXiv: https://arxiv.org/abs/2308.04332 · HTML: https://mlsanity.com/paper/2308.04332/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 38 references that are in the map.
- First posted on arXiv 8 Aug 2023.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [The Off-Switch Game](https://mlsanity.com/paper/1611.08219/) (2016)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference](https://mlsanity.com/paper/1906.09624/) (2019)
- [Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation](https://mlsanity.com/paper/2006.14804/) (2020)
- [A Survey on Interactive Reinforcement Learning: Design Principles and Open Challenges](https://mlsanity.com/paper/2105.12949/) (2021)
- [imitation: Clean Imitation Learning Implementations](https://mlsanity.com/paper/2211.11972/) (2022)
- [The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types](https://mlsanity.com/paper/2208.10687/) (2022)
- [How to talk so AI will learn: Instructions, descriptions, and autonomy](https://mlsanity.com/paper/2206.07870/) (2022)
- [Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning](https://mlsanity.com/paper/2206.13316/) (2022)
- [Unified Learning from Demonstrations, Corrections, and Preferences during Physical Human-Robot Interaction](https://mlsanity.com/paper/2207.03395/) (2022)
- [APReL: A Library for Active Preference-based Reward Learning Algorithms](https://mlsanity.com/paper/2108.07259/) (2021)
- [Reward Learning from Narrated Demonstrations](https://mlsanity.com/paper/1804.10692/) (2018)
- [Linguistic communication as (inverse) reward design](https://mlsanity.com/paper/2204.05091/) (2022)
- [Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym](https://mlsanity.com/paper/2102.02639/) (2021)

## Built on by (most cited first, 15 of 15)

- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [Optimal Design for Reward Modeling in RLHF](https://mlsanity.com/paper/2410.17055/) (2024)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [CREW: Facilitating Human-AI Teaming Research](https://mlsanity.com/paper/2408.00170/) (2024)
- [Reward Learning from Multiple Feedback Types](https://mlsanity.com/paper/2502.21038/) (2025)
- [Deconstructing Human-AI Collaboration: Agency, Interaction, and Adaptation](https://mlsanity.com/paper/2404.12056/) (2024)
- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2507.04340/) (2025)
- [DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition](https://mlsanity.com/paper/2507.18802/) (2025)
- [Improving User Interface Generation Models from Designer Feedback](https://mlsanity.com/paper/2509.16779/) (2025)
- [Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma](https://mlsanity.com/paper/2511.19504/) (2025)
- [Tell me why: Training preferences-based RL with human preferences and step-level explanations](https://mlsanity.com/paper/2405.14244/) (2024)
- [On the Effect of Robot Errors on Human Teaching Dynamics](https://mlsanity.com/paper/2409.09827/) (2024)
- [Building Intelligent User Interfaces for Human-AI Alignment](https://mlsanity.com/paper/2602.11753/) (2026)
- [MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inference](https://mlsanity.com/paper/2602.15206/) (2026)

## Abstract (arXiv)

To use reinforcement learning from human feedback (RLHF) in practical applications, it is crucial to learn reward models from diverse sources of human feedback and to consider human factors involved in providing feedback of different types. However, the systematic study of learning from diverse types of feedback is held back by limited standardized tooling available to researchers. To bridge this gap, we propose RLHF-Blender, a configurable, interactive interface for learning from human feedback. RLHF-Blender provides a modular experimentation framework and implementation that enables researchers to systematically investigate the properties and qualities of human feedback for reward learning. The system facilitates the exploration of various feedback types, including demonstrations, rankings, comparisons, and natural language instructions, as well as studies considering the impact of human factors on their effectiveness. We discuss a set of concrete research opportunities enabled by RLHF-Blender. More information is available at https://rlhfblender.info/.