mlsanityData to 14 Oct 2026

The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization

arXiv 2311.04919 ↗First posted 2 Nov 2023cs.CL cs.AI cs.HCcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) can be used to capture complex and nuanced properties of text generation quality. As a result, the task of text summarization has been identified as a good candidate for this process. In this paper, we explore how preference agreement impacts the efficacy of RLHF for summarization. We show that sampling human preferences to include a range of annotator agreement results in (1) higher accuracy reward models and (2) alters the characteristics of quality captured. We additionally show improvements in downstream generation when using a reward model trained with a range of preference agreements. Our contributions have implications for the design of synthetic datasets as well as the importance of considering quality differentials in comparison-based data.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 7 of its 23 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

7 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Learning to summarize from human feedback2 Sep

Other references · 6

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  5. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  6. 2015Statistical modality tagging from rule-based annotations and crowdsourcingcited by 6

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  2. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  3. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  4. 2025Reranking-based Generation for Unbiased Perspective Summarizationcited by 2
  5. 2025TaP: A Taxonomy-Guided Framework for Automated and Scalable Preference Data Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning to summarize user information for personalized reinforcement learning from human feedback
  2. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization
  5. 2026Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback
  6. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  7. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  8. 2022Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback