mlsanityData to 14 Oct 2026

RRHF: Rank Responses to Align Language Models with Human Feedback without tears

arXiv 2304.05302 ↗First posted 11 Apr 2023cs.CLcited by 463

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), reward model training, and Proximal Policy Optimization (PPO). However, PPO is sensitive to hyperparameters and requires multiple models in its standard implementation, making it hard to train and scale up to larger parameter counts. In contrast, we propose a novel learning paradigm called RRHF, which scores sampled responses from different sources via a logarithm of conditional probabilities and learns to align these probabilities with human preferences through ranking loss. RRHF can leverage sampled responses from various sources including the model responses from itself, other large language model responses, and human expert responses to learn to rank them. RRHF only needs 1 to 2 models during tuning and can efficiently align language models with human preferences robustly without complex hyperparameter tuning. Additionally, RRHF can be considered an extension of SFT and reward model training while being simpler than PPO in terms of coding, model counts, and hyperparameters. We evaluate RRHF on the Helpful and Harmless dataset, demonstrating comparable alignment performance with PPO by reward model score and human labeling. Extensive experiments show that the performance of RRHF is highly related to sampling quality which suggests RRHF is a best-of-n learner. Codes available at https://github.com/GanjinZero/RRHF.

Key facts. Cited by 463 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 58 references that are in the map. First posted on arXiv 11 Apr 2023.

Builds on

43 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers10 Feb
  2. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  4. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
24 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  10. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  11. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  12. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  13. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  14. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 301
  15. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  16. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  17. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
  18. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  19. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  20. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  21. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  22. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  23. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 77
  24. 2022HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 87; 2024: 190; 2025: 133; 2026: 53.

463 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  9. 2024Self-Rewarding Language Modelscited by 681
  10. 2024A Survey on Large Language Models for Code Generationcited by 658
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  12. 2023A Comprehensive Overview of Large Language Modelscited by 648
38 more
  1. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  2. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  6. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  8. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  9. 2023Large Language Model Unlearningcited by 302
  10. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  12. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  13. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  14. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  15. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  16. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  17. 2023Large Language Model Alignment: A Surveycited by 243
  18. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  19. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  20. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  21. 2023Mitigating the Alignment Tax of RLHFcited by 183
  22. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  23. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  24. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  25. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  26. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  27. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  28. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  29. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  30. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
  31. 2024Token-level Direct Preference Optimizationcited by 135
  32. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  33. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  34. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
  35. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  36. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 117
  37. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  38. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112

50 most cited of 463 All 463 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  2. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  4. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  5. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  6. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  7. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  8. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization