mlsanityData to 14 Oct 2026

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

arXiv 2204.05862 ↗First posted 12 Apr 2022cs.CL cs.LGcited by 3,720

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.

Key facts. Cited by 3,720 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 72 references that are in the map. First posted on arXiv 12 Apr 2022.

Builds on

50 of 72 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
37 more
  1. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  9. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  14. 2022Red Teaming Language Models with Language Modelscited by 1,026
  15. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  16. 2018Deep Anomaly Detection with Outlier Exposurecited by 942
  17. 2020Aligning AI With Shared Human Valuescited by 723
  18. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  19. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  20. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  21. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  22. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  23. 2021Unsolved Problems in ML Safetycited by 292
  24. 2022Teaching language models to support answers with verified quotescited by 292
  25. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  26. 2021Scaling Laws for Transfercited by 272
  27. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  28. 2022Predictability and Surprise in Large Generative Modelscited by 210
  29. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 207
  30. 2020Contrastive Training for Improved Out-of-Distribution Detectioncited by 186
  31. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  32. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  33. 2020Hybrid Models for Open Set Recognitioncited by 109
  34. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  35. 2019Hybrid Models with Deep and Invertible Featurescited by 41
  36. 2019Beyond Human-Level Accuracy: Computational Challenges in Deep Learningcited by 19
  37. 2022Adversarial vulnerability of powerful near out-of-distribution detectioncited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 38; 2023: 523; 2024: 1178; 2025: 1251; 2026: 730.

3,720 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  11. 2023A Survey of Large Language Modelscited by 2,576
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
38 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  12. 2023PaLM 2 Technical Reportcited by 1,162
  13. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  14. 2023LIMA: Less Is More for Alignmentcited by 1,100
  15. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  16. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  17. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  18. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  19. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  20. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  21. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  22. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  23. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  24. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  25. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  26. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  27. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  28. 2023Adversarial Diffusion Distillationcited by 759
  29. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  30. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  31. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  32. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  33. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  34. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  35. 2024Phi-4 Technical Reportcited by 711
  36. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  37. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  38. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686

50 most cited of 3,720 All 3,720 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023SuperHF: Supervised Iterative Learning from Human Feedback
  2. 2026The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
  3. 2026Reward Modeling from Natural Language Human Feedback
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  6. 2026A Regret Minimization Framework on Preference Learning in Large Language Models
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences