mlsanityData to 14 Oct 2026

RePO: Understanding Preference Learning Through ReLU-Based Optimization

arXiv 2503.07426 ↗First posted 10 Mar 2025cs.LG cs.AIcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences is critical for real-world deployment, yet existing methods like RLHF face computational and stability challenges. While DPO establishes an offline paradigm with single hyperparameter β, subsequent methods like SimPO reintroduce complexity through dual parameters (β, γ). We propose ReLU-based Preference Optimization (RePO), a streamlined algorithm that eliminates β via two advances: (1) retaining SimPO's reference-free margins but removing β through gradient analysis, and (2) adopting a ReLU-based max-margin loss that naturally filters trivial pairs. Theoretically, RePO is characterized as SimPO's limiting case (β → ∞), where the logistic weighting collapses to binary thresholding, forming a convex envelope of the 0-1 loss. Empirical results on AlpacaEval 2 and Arena-Hard show that RePO outperforms DPO and SimPO across multiple base models, requiring only one hyperparameter to tune.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 66 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

52 of 66 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  2. 2024β-DPO: Direct Preference Optimization with Dynamic β11 Jul
  3. 2024Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization10 Jul
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024sDPO: Don't Use Your Data All at Once28 Mar
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  8. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  9. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  10. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models15 Feb
  11. 2024Active Preference Learning for Large Language Models12 Feb
  12. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
5 more
  1. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 35

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2017Deep reinforcement learning from human preferencescited by 3,912
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
23 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  6. 2024Self-Rewarding Language Modelscited by 681
  7. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  11. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  13. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  14. 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 252
  15. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  16. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  17. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  18. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  19. 2024GLoRe: When, Where, and How to Improve LLM Reasoning via Global and Local Refinementscited by 103
  20. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  21. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  22. 2024Reward Difference Optimization For Sample Reweighting In Offline RLHFcited by 10
  23. 2024Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025On Symmetric Losses for Robust Policy Optimization with Noisy Preferencescited by 4
  2. 2025On Efficiency-Effectiveness Trade-off of Diffusion-based Recommenderscited by 3
  3. 2026Metaphors are a Source of Cross-Domain Misalignment of Large Reasoning Modelscited by 0
  4. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samplescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Intrinsic Mutual Information as a Modulator for Preference Optimization
  2. 2024Multi-Reference Preference Optimization for Large Language Models
  3. 2024MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples
  4. 2024iREPO: implicit Reward Pairwise Difference based Empirical Preference Optimization
  5. 2024Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
  6. 2024New Desiderata for Direct Preference Optimization
  7. 2025A Survey of Direct Preference Optimization
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models