mlsanityData to 14 Oct 2026

Fusing Rewards and Preferences in Reinforcement Learning

arXiv 2508.11363 ↗First posted 15 Aug 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

We present Dual-Feedback Actor (DFA), a reinforcement learning algorithm that fuses both individual rewards and pairwise preferences (if available) into a single update rule. DFA uses the policy's log-probabilities directly to model the preference probability, avoiding a separate reward-modeling step. Preferences can be provided by human-annotators (at state-level or trajectory-level) or be synthesized online from Q-values stored in an off-policy replay buffer. Under a Bradley-Terry model, we prove that minimizing DFA's preference loss recovers the entropy-regularized Soft Actor-Critic (SAC) policy. Our simulation results show that DFA trained on generated preferences matches or exceeds SAC on six control environments and demonstrates a more stable training process. With only a semi-synthetic preference dataset under Bradley-Terry model, our algorithm outperforms reward-modeling reinforcement learning from human feedback (RLHF) baselines in a stochastic GridWorld and approaches the performance of an oracle with true rewards.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 41 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

26 of 41 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference25 Sep
  2. 2024Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis11 Jun
  3. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF31 May
  4. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  5. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  6. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  7. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  7. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  8. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  10. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
4 more
  1. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 81
  2. 2020Preference-based Reinforcement Learning with Finite-Time Guaranteescited by 65
  3. 2024Soft Preference Optimization: Aligning Language Models to Expert Distributionscited by 9
  4. 2025Length-Controlled Margin-Based Preference Optimization without Reference Modelcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  2. 2025Repairing Reward Functions with Feedback to Mitigate Reward Hacking
  3. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO
  6. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  7. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
  8. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design