mlsanityData to 7 Oct 2026

Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback

arXiv 2402.02423 ↗First posted 4 Feb 2024cs.LG cs.AI cs.HCcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) has received significant attention for performing tasks without the need for costly manual reward design by aligning human preferences. It is crucial to consider diverse human feedback types and various learning methods in different environments. However, quantifying progress in RLHF with diverse feedback is challenging due to the lack of standardized annotation platforms and widely used unified benchmarks. To bridge this gap, we introduce Uni-RLHF, a comprehensive system implementation tailored for RLHF. It aims to provide a complete workflow from real human feedback, fostering progress in the development of practical problems. Uni-RLHF contains three packages: 1) a universal multi-feedback annotation platform, 2) large-scale crowdsourced feedback datasets, and 3) modular offline RLHF baseline implementations. Uni-RLHF develops a user-friendly annotation interface tailored to various feedback types, compatible with a wide range of mainstream RL environments. We then establish a systematic pipeline of crowdsourced annotations, resulting in large-scale annotated datasets comprising more than 15 million steps across 30+ popular tasks. Through extensive experiments, the results in the collected datasets demonstrate competitive performance compared to those from well-designed manual rewards. We evaluate various design choices and offer insights into their strengths and potential areas of improvement. We wish to build valuable open-source platforms, datasets, and baselines to facilitate the development of more robust and reliable RLHF solutions based on realistic human feedback. The website is available at https://uni-rlhf.github.io/.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 77 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

55 of 77 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback8 Aug
  2. 2023Primitive Skill-based Robot Learning from Human Evaluative Feedback28 Jul
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2023Reinforcement Learning from Diverse Human Preferences27 Jan
  5. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learning3 Jan
  6. 2022The Expertise Problem: Learning from Specialized Feedback12 Nov
  7. 2022Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences28 Oct
  8. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types23 Aug
  9. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  11. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
3 more
  1. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  2. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  10. 2017Inverse Reward Designcited by 246
  11. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  12. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
28 more
  1. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  2. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  4. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  5. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  6. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  7. 2022Differentiable Integrated Motion Prediction and Planning with Learnable Cost Function for Autonomous Drivingcited by 71
  8. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  9. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  10. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  11. 2019Exploratory Not Explanatory: Counterfactual Analysis of Saliency Maps for Deep Reinforcement Learningcited by 60
  12. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  13. 2022ProMix: Combating Label Noise via Maximizing Clean Sample Utilitycited by 48
  14. 2015DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixationscited by 47
  15. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  16. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  17. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  18. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  19. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  20. 2019Learning a Decision Module by Imitating Driver's Control Behaviorscited by 17
  21. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  22. 2021APReL: A Library for Active Preference-based Reward Learning Algorithmscited by 14
  23. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 14
  24. 2023Rating-based Reinforcement Learningcited by 14
  25. 2023Efficient RL via Disentangled Environment and Agent Representationscited by 11
  26. 2018Active Deep Q-learning with Demonstrationcited by 10
  27. 2023Learning to Identify Critical States for Reinforcement Learning from Videoscited by 9
  28. 2020DERAIL: Diagnostic Environments for Reward And Imitation Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 9; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  3. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  4. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Surveycited by 28
  5. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  6. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  7. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11
  8. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  9. 2025From Chaos to Order: The Atomic Reasoner Framework for Fine-grained Reasoning in Large Language Modelscited by 5
  10. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  11. 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 2
  12. 2025Robust Reward Alignment via Hypothesis Space Batch Cuttingcited by 2
8 more
  1. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learningcited by 2
  2. 2024On the Effect of Robot Errors on Human Teaching Dynamicscited by 0
  3. 2024LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiencycited by 0
  4. 2025Integrating Human Feedback into a Reinforcement Learning-Based Framework for Adaptive User Interfacescited by 0
  5. 2026MAVRL: Learning Reward Functions from Multiple Feedback Types with Amortized Variational Inferencecited by 0
  6. 2026From Reward-Free Representations to Preferences: Rethinking Offline Preference-Based Reinforcement Learningcited by 0
  7. 2026ViTaR: Visuo-Tactile Residual Adaptation for Foundation VLA Manipulationcited by 0
  8. 2026Subspace Inference Enables Efficient Active Reward Learning from Preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey of Reinforcement Learning from Human Feedback
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  4. 2025Reinforcement Learning from Human Feedback
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024Parameter Efficient Reinforcement Learning from Human Feedback
  7. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
  8. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning