mlsanityData to 7 Oct 2026

A Critical Evaluation of AI Feedback for Aligning Large Language Models

arXiv 2402.12366 ↗First posted 19 Feb 2024cs.LG cs.AI cs.CLcited by 36

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning with AI feedback (RLAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing RLAIF pipelines. More generally, we find that the gains from RLAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step RLAIF pipeline as well as suggestions for making RLAIF maximally useful in practice.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 47 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

27 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2023Mistral 7Bcited by 3,065
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2020Learning to summarize from human feedbackcited by 2,562
9 more
  1. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  2. 2024Mixtral of Expertscited by 1,785
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  6. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  7. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  8. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  9. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 18; 2025: 12; 2026: 6.

36 papers.

Sort:

By area:

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  3. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  4. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  5. 2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
  6. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  7. 2024Personality Alignment of Large Language Modelscited by 36
  8. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  9. 2024Reinforcement Learning from Multi-role Debates as Feedback for Bias Mitigation in LLMscited by 22
  10. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Modelscited by 20
  11. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planningcited by 19
  12. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 19
24 more
  1. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 18
  2. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  3. 2025CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Modelscited by 14
  4. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  5. 2024Dialogue Action Tokens: Steering Language Models in Goal-Directed Dialogue with a Multi-Turn Plannercited by 12
  6. 2024Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Modelscited by 12
  7. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 11
  8. 2024Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generationcited by 7
  9. 2025RLTHF: Targeted Human Feedback for LLM Alignmentcited by 7
  10. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  11. 2026DeepInnovator: Triggering the Innovative Capabilities of LLMscited by 4
  12. 2024On The Global Convergence Of Online RLHF With Neural Parametrizationcited by 3
  13. 2024Data-Centric Improvements for Enhancing Multi-Modal Understanding in Spoken Conversation Modelingcited by 3
  14. 2025Research Superalignment Should Advance Now with Alternating Competence and Conformity Optimizationcited by 1
  15. 2025Understanding Reinforcement Learning for Model Training, and future directions with GRAPEcited by 1
  16. 2025How well can LLMs provide planning feedback in grounded environments?cited by 1
  17. 2026Mitigating Mismatch within Reference-based Preference Optimizationcited by 1
  18. 2026On the Convergence of Self-Improving Online LLM Alignmentcited by 1
  19. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  20. 2025Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Modelscited by 0
  21. 2025Zero-shot 3D Map Generation with LLM Agents: A Dual-Agent Architecture for Procedural Content Generationcited by 0
  22. 2026LLM Active Alignment: A Nash Equilibrium Perspectivecited by 0
  23. 2026On Distinguishing Capability Elicitation from Capability Creation in Post-Training: A Free-Energy Perspectivecited by 0
  24. 2026VSPO: Vector-Steered Policy Optimization for Behavioral Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  2. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
  3. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback
  4. 2025Checklists Are Better Than Reward Models For Aligning Language Models
  5. 2024ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedback
  6. 2023Interpreting Learned Feedback Patterns in Large Language Models
  7. 2023Fine-tuning Language Models with Generative Adversarial Reward Modelling
  8. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers