mlsanityData to 7 Oct 2026

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

arXiv 2403.08309 ↗First posted 13 Mar 2024cs.LG cs.AIcited by 15

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from AI Feedback (RLAIF) has the advantages of shorter annotation cycles and lower costs over Reinforcement Learning from Human Feedback (RLHF), making it highly efficient during the rapid strategy iteration periods of large language model (LLM) training. Using ChatGPT as a labeler to provide feedback on open-domain prompts in RLAIF training, we observe an increase in human evaluators' preference win ratio for model responses, but a decrease in evaluators' satisfaction rate. Analysis suggests that the decrease in satisfaction rate is mainly due to some responses becoming less helpful, particularly in terms of correctness and truthfulness, highlighting practical limitations of basic RLAIF. In this paper, we propose Hybrid Reinforcement Learning from AI Feedback (HRLAIF). This method enhances the accuracy of AI annotations for responses, making the model's helpfulness more robust in training process. Additionally, it employs AI for Red Teaming, further improving the model's harmlessness. Human evaluation results show that HRLAIF inherits the ability of RLAIF to enhance human preference for outcomes at a low cost while also improving the satisfaction rate of responses. Compared to the policy model before Reinforcement Learning (RL), it achieves an increase of 2.08% in satisfaction rate, effectively addressing the issue of a decrease of 4.58% in satisfaction rate after basic RLAIF.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 47 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

31 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
15 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  9. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  10. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  11. 2015LCSTS: A Large Scale Chinese Short Text Summarization Datasetcited by 65
  12. 2023Chinese Open Instruction Generalist: A Preliminary Releasecited by 35
  13. 2019Generating Empathetic Responses by Looking Ahead the User's Sentimentcited by 27
  14. 2023An Empirical Study of Instruction-tuning Large Language Models in Chinesecited by 17
  15. 2021Training Value-Aligned Reinforcement Learning Agents Using a Normative Priorcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 4.

15 papers.

Sort:

By area:

  1. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  2. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  3. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  5. 2024Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Modelscited by 11
  6. 2026Reason Only When Needed: Efficient Generative Reward Modeling via Model-Internal Uncertaintycited by 6
  7. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  8. 2025Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHFcited by 5
  9. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
  10. 2025OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responsescited by 3
  11. 2026Order Matters in Retrosynthesis: Structure-aware Generation via Reaction-Center-Guided Discrete Flow Matchingcited by 3
  12. 2025Self Iterative Label Refinement via Robust Unlabeled Learningcited by 2
3 more
  1. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 1
  2. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Mergingcited by 1
  3. 2024An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  2. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  3. 2024Multi-objective Reinforcement learning from AI Feedback
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  5. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  6. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback
  7. 2023A Survey of Reinforcement Learning from Human Feedback
  8. 2024Direct Language Model Alignment from Online AI Feedback