mlsanityData to 7 Oct 2026

Fine-Tuning Language Models from Human Preferences

arXiv 1909.08593 ↗First posted 18 Sep 2019cs.CL cs.LG stat.MLcited by 2,124

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.

Key facts. Cited by 2,124 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 53 references that are in the map. First posted on arXiv 18 Sep 2019.

Builds on

39 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  2. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  3. 2018Learning to Understand Goal Specifications by Modelling Reward5 Jun
  4. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning27 May
  5. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2015Image-based Recommendations on Styles and Substitutescited by 702
  11. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  12. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
22 more
  1. 2018AI safety via debatecited by 384
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  3. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  4. 2015Semi-supervised Sequence Learningcited by 198
  5. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  6. 2018Bottom-Up Abstractive Summarizationcited by 177
  7. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  8. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  9. 2018Supervising strong learners by amplifying weak expertscited by 139
  10. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  11. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  12. 2019Discriminative Active Learningcited by 97
  13. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  14. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  15. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  16. 2016Dialogue Learning With Human-In-The-Loopcited by 46
  17. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  18. 2019Sample Efficient Text Summarization Using a Single Pre-Trained Transformercited by 27
  19. 2016Active Learning for Speech Recognition: the Power of Gradientscited by 25
  20. 2019Finding Generalizable Evidence by Learning to Convince Q&A Modelscited by 25
  21. 2019Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisationcited by 14
  22. 2019Preference-based Interactive Multi-Document Summarisationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 4; 2021: 42; 2022: 75; 2023: 361; 2024: 632; 2025: 628; 2026: 381.

2,124 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
38 more
  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  3. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  7. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  8. 2023A Survey on Multimodal Large Language Modelscited by 898
  9. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  10. 2023Steering Language Models With Activation Engineeringcited by 824
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  13. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  14. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  15. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  16. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  17. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  18. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  19. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  20. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  21. 2024Self-Rewarding Language Modelscited by 654
  22. 2023A Comprehensive Overview of Large Language Modelscited by 639
  23. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  24. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  25. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  26. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  27. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  28. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  29. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
  30. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  31. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  32. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  33. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  34. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 442
  35. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  36. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  37. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  38. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401

50 most cited of 2,124 All 2,124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAM: A Generative Foundation Reward Model for Reward Generalization
  2. 2026Reward Modeling from Natural Language Human Feedback
  3. 2025Sentence-level Reward Model can Generalize Better for Aligning LLM from Human Preference
  4. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  5. 2025PIRA: Preference-Oriented Instruction-Tuned Reward Models with Dual Aggregation
  6. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systems
  7. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  8. 2026RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences