mlsanityData to 7 Oct 2026

ARF-RLHF: Adaptive Reward-Following for RLHF through Emotion-Driven Self-Supervision and Trace-Biased Dynamic Optimization

arXiv 2507.03069 ↗First posted 3 Jul 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Current RLHF methods such as PPO and DPO typically reduce human preferences to binary labels, which are costly to obtain and too coarse to reflect individual variation. We observe that expressions of satisfaction and dissatisfaction follow stable linguistic patterns across users, indicating that more informative supervisory signals can be extracted from free-form feedback. Building on this insight, we introduce Adaptive Reward-Following (ARF), which converts natural feedback into continuous preference trajectories and optimizes them using the novel TraceBias algorithm. Across diverse LLMs and preference domains, ARF consistently outperforms PPO and DPO, improving alignment by up to 7.6%. Our results demonstrate that continuous reward modeling provides a scalable path toward personalized and theoretically grounded RLHF.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 40 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

15 of 40 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  9. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  10. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  11. 2023Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversationscited by 88
  12. 2024PLaD: Preference-based Large Language Model Distillation with Pseudo-Preference Pairscited by 11
1 more
  1. 2017Linguistic Markers of Influence in Informal Interactionscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PILAF: Optimal Human Preference Sampling for Reward Modeling
  2. 2025Influencing Humans to Conform to Preference Models for RLHF
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization
  5. 2023The History and Risks of Reinforcement Learning and Human Feedback
  6. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  7. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHF
  8. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback