mlsanityData to 7 Oct 2026

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

arXiv 2608.15530 ↗First posted 16 Aug 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning with human feedback (RLHF) aligns LLMs with human preferences, improving summarization fluency and safety, but causes sentiment drift: overly neutral summaries stripped of emotional nuance. We diagnose why RL acts as a sentiment neutralizer and present Policy Attribution, a framework using gradient and logit decomposition to trace drift to reward model (RM) signals and KL (Kullback-Leibler) penalty. Sentiment drift reflects a strategic bias toward "low-risk" tokens maximizing expected rewards under preference uncertainty (Stiennon et al., 2020; Gao, Schulman, and Hilton, 2023). On Reddit TL;DR and CNN/DailyMail, RLHF summaries get higher rewards but show 30-40% lower sentiment variance. Cross-lingual analysis across eight languages shows language-independent drift, with morphologically richer languages more suppressed (Krasitskii et al., 2026). We propose and validate a sentiment-aware regularization technique reducing drift by 18-22% without harming summary quality. The code and toolkit will be public.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 8 of its 18 references that are in the map. First posted on arXiv 16 Aug 2026.

Builds on

8 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHF7 Nov
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 6

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  5. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 246
  6. 2025Multilingual Sentiment Analysis of Summarized Texts: A Cross-Language Study of Text Shortening Effectscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance
  2. 2023The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization
  3. 2020Learning to summarize from human feedback
  4. 2026Rater State Bias in RLHF Preference Data: An Audit Framework
  5. 2024Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  7. 2024From Lists to Emojis: How Format Bias Affects Model Alignment
  8. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases