mlsanityData to 7 Oct 2026

Learning to summarize from human feedback

arXiv 2009.01325 ↗First posted 2 Sep 2020cs.CL cs.AI cs.LGcited by 2,562

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

Key facts. Cited by 2,562 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 83 references that are in the map. First posted on arXiv 2 Sep 2020.

Builds on

50 of 83 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback3 May

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
37 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  2. 2015Teaching Machines to Read and Comprehendcited by 1,039
  3. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  5. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  6. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  8. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  9. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  10. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  13. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  14. 2018The Natural Language Decathlon: Multitask Learning as Question Answeringcited by 235
  15. 2020ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-trainingcited by 221
  16. 2015Semi-supervised Sequence Learningcited by 198
  17. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  18. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  19. 2016An Actor-Critic Algorithm for Sequence Predictioncited by 158
  20. 2018Supervising strong learners by amplifying weak expertscited by 139
  21. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  22. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  23. 2019Learning from Dialogue after Deployment: Feed Yourself, Chatbot!cited by 102
  24. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  25. 2020Trading Off Diversity and Quality in Natural Language Generationcited by 94
  26. 2019ACUTE-EVAL: Improved Dialogue Evaluation with Optimized Questions and Multi-turn Comparisonscited by 83
  27. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  28. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  29. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  30. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 63
  31. 2018The price of debiasing automatic metrics in natural language evaluationcited by 60
  32. 2018Polite Dialogue Generation Without Parallel Datacited by 57
  33. 2018BanditSum: Extractive Summarization as a Contextual Banditcited by 46
  34. 2019Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluatorscited by 34
  35. 2020Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Modelscited by 27
  36. 2019Finding Generalizable Evidence by Learning to Convince Q&A Modelscited by 25
  37. 2019Reward Learning for Efficient Reinforcement Learning in Extractive Document Summarisationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 39; 2022: 99; 2023: 440; 2024: 723; 2025: 722; 2026: 538.

2,562 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2023A Survey of Large Language Modelscited by 2,537
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
38 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  11. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  13. 2023A Survey on Multimodal Large Language Modelscited by 898
  14. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  15. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  16. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  17. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  18. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  19. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  20. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  21. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  22. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  23. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  24. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  25. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  26. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  27. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
  28. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  29. 2024Self-Rewarding Language Modelscited by 654
  30. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  31. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  32. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  33. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  34. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  35. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  36. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  37. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  38. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556

50 most cited of 2,562 All 2,562 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AlignSum: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preference
  2. 2024How Much Annotation is Needed to Compare Summarization Models?
  3. 2022Self-critiquing models for assisting human evaluators
  4. 2026Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback
  5. 2022Towards Abstractive Timeline Summarisation using Preference-based Reinforcement Learning
  6. 2024Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization
  7. 2025Learning from Self Critique and Refinement for Faithful LLM Summarization
  8. 2025Balancing Rewards in Text Summarization: Multi-Objective Reinforcement Learning via HyperVolume Optimization