mlsanityData to 14 Oct 2026

DDO-RM: Distribution-Level Policy Improvement after Reward Learning

arXiv 2604.11119 ↗First posted 13 Apr 2026stat.ML cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Recent theory suggests that reward-model-first methods can be more sample-efficient than direct policy fitting when the reward function is statistically simpler than the induced policy. We propose DDO-RM, a finite-candidate decision-optimization method that converts reward scores into an explicit target distribution. Unlike PPO-based RLHF or DPO, DDO-RM performs a KL-regularized mirror-descent update to project the policy toward a reward-improved distribution over a candidate set. Preliminary experiments on Pythia-410M show that DDO-RM outperforms DPO in pair accuracy (0.52 to 0.56) and mean margin (0.13 to 0.53). Our framework provides a principled connection between reward learning and mirror-descent policy improvement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 8 of its 18 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

8 of 18 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO26 May
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 6

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  6. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
  2. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Models
  3. 2026Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
  4. 2025Value-Free Policy Optimization via Reward Partitioning
  5. 2026GOPO: Policy Optimization using Ranked Rewards
  6. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Data
  7. 2025Reward Models in Deep Reinforcement Learning: A Survey
  8. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference