mlsanityData to 14 Oct 2026

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

arXiv 2309.00267 ↗First posted 1 Sep 2023cs.CL cs.AI cs.LGcited by 625

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards "self-improvement" by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.

Key facts. Cited by 625 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 54 references that are in the map. First posted on arXiv 1 Sep 2023.

Builds on

44 of 54 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2020Learning to summarize from human feedbackcited by 2,588
  12. 2016Concrete Problems in AI Safetycited by 1,642
26 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  4. 2018Hierarchical Neural Story Generationcited by 1,193
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2022Large Language Models Can Self-Improvecited by 664
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  10. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  12. 2019A Theory of Regularized Markov Decision Processescited by 286
  13. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  14. 2023Reward Design with Language Modelscited by 261
  15. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  16. 2022Is GPT-3 a Good Data Annotator?cited by 208
  17. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  18. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 127
  19. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  20. 2021Towards Zero-Label Language Learningcited by 76
  21. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  22. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  23. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  24. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  25. 2016Avoiding Wireheading with Value Reinforcement Learningcited by 15
  26. 2019Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 39; 2024: 174; 2025: 217; 2026: 195.

625 papers.

Sort:

By area:

  1. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  4. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  5. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  6. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  7. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  9. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  10. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  11. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  12. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
38 more
  1. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 241
  2. 2023Nash Learning from Human Feedbackcited by 224
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  5. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  6. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  7. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  8. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 148
  9. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  10. 2025Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tailcited by 146
  11. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  12. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  13. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  14. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  15. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  16. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  17. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  18. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  19. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  20. 2024Critique-out-Loud Reward Modelscited by 106
  21. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 98
  22. 2024Aligner: Efficient Alignment by Learning to Correctcited by 98
  23. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 91
  24. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  25. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  26. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  27. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  28. 2025Self-Adapting Language Modelscited by 84
  29. 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 82
  30. 2025Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?cited by 81
  31. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  32. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedbackcited by 80
  33. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 78
  34. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  35. 2023AceGPT, Localizing Large Language Models in Arabiccited by 75
  36. 2024Intention Analysis Makes LLMs A Good Jailbreak Defendercited by 74
  37. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  38. 2024Self-Taught Evaluatorscited by 72

50 most cited of 625 All 625 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback
  4. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models
  5. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
  7. 2024Generative Reward Models
  8. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback