Data to 14 Oct 2026

REBEL: Reinforcement Learning via Regressing Relative Rewards

arXiv 2404.16767 ↗First posted 25 Apr 2024cs.LG cs.CL cs.CVcited by 69

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more

Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

While originally developed for continuous control problems, Proximal Policy Optimization (PPO) has emerged as the work-horse of a variety of reinforcement learning (RL) applications, including the fine-tuning of generative models. Unfortunately, PPO requires multiple heuristics to enable stable convergence (e.g. value networks, clipping), and is notorious for its sensitivity to the precise implementation of these components. In response, we take a step back and ask what a minimalist RL algorithm for the era of generative models would look like. We propose REBEL, an algorithm that cleanly reduces the problem of policy optimization to regressing the relative reward between two completions to a prompt in terms of the policy, enabling strikingly lightweight implementation. In theory, we prove that fundamental RL algorithms like Natural Policy Gradient can be seen as variants of REBEL, which allows us to match the strongest known theoretical guarantees in terms of convergence and sample complexity in the RL literature. REBEL can also cleanly incorporate offline data and be extended to handle the intransitive preferences we frequently see in practice. Empirically, we find that REBEL provides a unified approach to language modeling and image generation with stronger or similar performance as PPO and DPO, all while being simpler to implement and more computationally efficient than PPO. When fine-tuning Llama-3-8B-Instruct, REBEL achieves strong performance in AlpacaEval 2.0, MT-Bench, and Open LLM Leaderboard.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 73 of its 122 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

73 of 122 references.

Closest ancestors · 13

Same direction, newest first

1 more

2017Deep reinforcement learning from human preferences12 Jun

Other references · 60

2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2020Measuring Massive Multitask Language Understandingcited by 7,963
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975

38 more

2020Learning to summarize from human feedbackcited by 2,588
2015Trust Region Policy Optimizationcited by 2,580
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
2023Consistency Modelscited by 1,822
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
2023Training Diffusion Models with Reinforcement Learningcited by 982
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
2017Deep Reinforcement Learning that Matterscited by 754
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
2023Aligning Text-to-Image Models using Human Feedbackcited by 422
2023Efficient Online Reinforcement Learning with Offline Datacited by 363
2024Direct Language Model Alignment from Online AI Feedbackcited by 260
2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 246
2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 118
2019Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPscited by 118
2015Contextual Dueling Banditscited by 101
2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 92
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gapcited by 81
2020VarGrad: A Low-Variance Gradient Estimator for Variational Inferencecited by 75
2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
2021Efficient First-Order Contextual Bandits: Prediction, Allocation, and Triangular Discriminationcited by 53
2023Learning to Generate Better Than Your LLMcited by 49
2023Fine-Tuning Language Models with Advantage-Induced Policy Alignmentcited by 48
2016Learning to Perform Physics Experiments via Deep Reinforcement Learningcited by 47
2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 44

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 28; 2026: 20.

69 papers.

Sort:

By area:

2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
2024Robust Preference Optimization through Reward Model Distillationcited by 67
2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
2024The Importance of Online Data: Understanding Preference Fine-tuning via Coveragecited by 55
2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48

38 more

2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 30
2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 27
2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
2025Direct Alignment with Heterogeneous Preferencescited by 19
2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
2025Token-Importance Guided Direct Preference Optimizationcited by 14
2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
2024End-to-end Training for Recommendation with Language-based User Profilescited by 10
2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
2024The Central Role of the Loss Function in Reinforcement Learningcited by 8
2024COMAL: A Convergent Meta-Algorithm for Aligning LLMs with General Preferencescited by 8
2025Diffusion-Sharpening: Fine-tuning Diffusion Models with Denoising Trajectory Sharpeningcited by 7
2024Humanizing the Machine: Proxy Attacks to Mislead LLM Detectorscited by 6
2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Explorationcited by 6
2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
2026Coverage Improvement and Fast Convergence of On-policy Preference Learningcited by 6
2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learningcited by 5
2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
2026Diffusion Reinforcement Learning via Centered Reward Distillationcited by 5
2026HPSv3++: Scaling Reward Models Across the Full Spectrum of Diffusion Model Capabilitiescited by 5
2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontiercited by 4
2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspectivecited by 4
2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
2025SquarechiPO: Differentially Private and Robust chi2-Preference Optimization in Offline Direct Alignmentcited by 3
2025Beyond RLHF: A Unified Theoretical Framework of Alignmentcited by 3
2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 3
2025MAS2: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systemscited by 3
2025Why DPO is a Misspecified Estimator and How to Fix Itcited by 3
2025On the Sample Complexity of Differentially Private Policy Optimizationcited by 3
2026Reward Models Inherit Value Biases from Pretrainingcited by 3
2025Refining Few-Step Text-to-Multiview Diffusion via Reinforcement Learningcited by 2
2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
2025Behavior Preference Regression for Offline Reinforcement Learningcited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way