Data to 14 Oct 2026

RLHF Workflow: From Reward Modeling to Online RLHF

arXiv 2405.07863 ↗First posted 13 May 2024cs.LG cs.AI cs.CLcited by 267

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more

LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. However, existing open-source RLHF projects are still largely confined to the offline learning setting. In this technical report, we aim to fill in this gap and provide a detailed recipe that is easy to reproduce for online iterative RLHF. In particular, since online human feedback is usually infeasible for open-source communities with limited resources, we start by constructing preference models using a diverse set of open-source datasets and use the constructed proxy preference model to approximate human feedback. Then, we discuss the theoretical insights and algorithmic principles behind online iterative RLHF, followed by a detailed practical implementation. Our trained LLM achieves impressive performance on LLM chatbot benchmarks, including AlpacaEval-2, Arena-Hard, and MT-Bench, as well as other academic benchmarks such as HumanEval and TruthfulQA. We have shown that supervised fine-tuning (SFT) and iterative RLHF can obtain state-of-the-art performance with fully open-source datasets. Further, we have made our models, curated datasets, and comprehensive step-by-step code guidebooks publicly available. Please refer to https://github.com/RLHFlow/RLHF-Reward-Modeling and https://github.com/RLHFlow/Online-RLHF for more detailed information.

Key facts. Cited by 267 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 69 of its 91 references that are in the map. First posted on arXiv 13 May 2024.

Builds on

69 of 91 references.

Closest ancestors · 4

Same direction, newest first

2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
2022Constitutional AI: Harmlessness from AI Feedback15 Dec
2017Proximal Policy Optimization Algorithms20 Jul

Other references · 65

2023GPT-4 Technical Reportcited by 18,396
2022Training language models to follow instructions with human feedbackcited by 15,971
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2021Evaluating Large Language Models Trained on Codecited by 8,320
2020Measuring Massive Multitask Language Understandingcited by 7,963
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2017Deep reinforcement learning from human preferencescited by 3,912
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2021Program Synthesis with Large Language Modelscited by 3,637

38 more

2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2020Learning to summarize from human feedbackcited by 2,588
2019Fine-Tuning Language Models from Human Preferencescited by 2,150
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
2023Instruction Tuning with GPT-4cited by 686
2024Self-Rewarding Language Modelscited by 681
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
2023Zephyr: Direct Distillation of LM Alignmentcited by 506
2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
2021Recursively Summarizing Books with Human Feedbackcited by 290
2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
2024Direct Language Model Alignment from Online AI Feedbackcited by 260
2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 186
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 62; 2025: 140; 2026: 65.

267 papers.

Sort:

By area:

2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 184
2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 145

38 more

2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cyclescited by 124
2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 112
2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
2024Critique-out-Loud Reward Modelscited by 106
2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
2025Self-rewarding correction for mathematical reasoningcited by 70
2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 68
2024Robust Preference Optimization through Reward Model Distillationcited by 67
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
2024Quantile Regression for Distributional Reward Models in RLHFcited by 60
2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 54
2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
2025SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Datacited by 51
2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
2024Entropy-Regularized Process Reward Modelcited by 38
2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
2025Scalable Chain of Thoughts via Elastic Reasoningcited by 34
2024Direct Judgement Preference Optimizationcited by 33
2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33

50 most cited of 267 All 267 (JSON)

Similar, not cited

Close in wording, no citation either way