Data to 7 Oct 2026

Statistical Rejection Sampling Improves Preference Optimization

arXiv 2309.06657 ↗First posted 13 Sep 2023cs.CLcited by 321

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized Reinforcement Learning from Human Feedback (RLHF) via online RL methods such as Proximal Policy Optimization (PPO). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the need for a separate reward model. However, the maximum likelihood estimator (MLE) of the target optimal policy requires labeled preference pairs sampled from that policy. DPO's lack of a reward model constrains its ability to sample preference pairs from the optimal policy, and SLiC is restricted to sampling preference pairs only from the SFT policy. To address these limitations, we introduce a novel approach called Statistical Rejection Sampling Optimization (RSO) that aims to source preference data from the target optimal policy using rejection sampling, enabling a more accurate estimation of the optimal policy. We also propose a unified framework that enhances the loss functions used in both SLiC and DPO from a preference modeling standpoint. Through extensive experiments across three diverse tasks, we demonstrate that RSO consistently outperforms both SLiC and DPO on evaluations from both Large Language Model (LLM) and human raters.

Key facts. Cited by 321 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 38 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

32 of 38 references.

Closest ancestors · 2

Same direction, newest first

2023Making Large Language Models Better Reasoners with Alignment5 Sep
2023Preference Ranking Optimization for Human Alignment30 Jun

Other references · 30

2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2022Training Compute-Optimal Large Language Modelscited by 2,927
2020Learning to summarize from human feedbackcited by 2,562

18 more

2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
2023PaLM 2 Technical Reportcited by 1,154
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2015Teaching Machines to Read and Comprehendcited by 1,039
2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
2022Efficiently Scaling Transformer Inferencecited by 529
2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 162; 2025: 104; 2026: 42.

321 papers.

Sort:

By area:

2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
2024Direct Language Model Alignment from Online AI Feedbackcited by 257
2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 238
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184

38 more

2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
2024Token-level Direct Preference Optimizationcited by 135
2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 133
2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
2024A Survey on Large Language Model-Based Game Agentscited by 106
2024Critique-out-Loud Reward Modelscited by 105
2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 103
2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
2024β-DPO: Direct Preference Optimization with Dynamic βcited by 89
2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 68
2024Decoding-time Realignment of Language Modelscited by 67
2025Self-rewarding correction for mathematical reasoningcited by 67
2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
2024On Softmax Direct Preference Optimization for Recommendationcited by 64
2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61

50 most cited of 321 All 321 (JSON)

Similar, not cited

Close in wording, no citation either way