mlsanityData to 7 Oct 2026

AlphaPO: Reward Shape Matters for LLM Alignment

arXiv 2501.03884 ↗First posted 7 Jan 2025cs.CLcited by 14

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning with Human Feedback (RLHF) and its variants have made huge strides toward the effective alignment of large language models (LLMs) to follow instructions and reflect human values. More recently, Direct Alignment Algorithms (DAAs) have emerged in which the reward modeling stage of RLHF is skipped by characterizing the reward directly as a function of the policy being learned. Some popular examples of DAAs include Direct Preference Optimization (DPO) and Simple Preference Optimization (SimPO). These methods often suffer from likelihood displacement, a phenomenon by which the probabilities of preferred responses are often reduced undesirably. In this paper, we argue that, for DAAs the reward (function) shape matters. We introduce AlphaPO, a new DAA method that leverages an α-parameter to help change the shape of the reward function beyond the standard log reward. AlphaPO helps maintain fine-grained control over likelihood displacement and over-optimization. Compared to SimPO, one of the best performing DAAs, AlphaPO leads to about 7% to 10% relative improvement in alignment performance for the instruct versions of Mistral-7B and Llama3-8B while achieving 15% to 50% relative improvement over DPO on the same models. The analysis and results presented highlight the importance of the reward shape and how one can systematically change it to affect training dynamics, as well as improve alignment performance.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 42 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

32 of 42 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithms15 Oct
  2. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization11 Oct
  3. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  4. 2024Robust Preference Optimization through Reward Model Distillation29 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  7. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  8. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  10. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  11. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  12. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  12. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
8 more
  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  2. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  3. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  4. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  5. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  6. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  7. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  8. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025On the Robustness of Reward Models for Language Model Alignmentcited by 13
  2. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  3. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  4. 2025The Differences Between Direct Alignment Algorithms are a Blurcited by 3
  5. 2025Learning Explainable Dense Reward Shapes via Bayesian Optimizationcited by 2
  6. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
  7. 2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
  8. 2026Normalized Rewards for Preference Optimizationcited by 1
  9. 2025DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)cited by 0
  10. 2025SSFO: Self-Supervised Faithfulness Optimization for Retrieval-Augmented Generationcited by 0
  11. 2025Failure Modes of Maximum Entropy RLHFcited by 0
  12. 2026Bayesian Preference Learning for Test-Time Steerable Reward Modelscited by 0
2 more
  1. 2026GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learningcited by 0
  2. 2026Masking Frequent Tokens Sharpens Direct Preference Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Bootstrapping Language Models with DPO Implicit Rewards
  2. 2025Towards Bridging the Reward-Generation Gap in Direct Alignment Algorithms
  3. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  4. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  5. 2025A Principled Loss Function for Direct Language Model Alignment
  6. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
  7. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
  8. 2024Direct Preference Optimization with an Offset