mlsanityData to 7 Oct 2026

RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

arXiv 2402.10038 ↗First posted 15 Feb 2024cs.CL cs.AI cs.CVcited by 48

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize the estimated reward during alignment. Recently, direct preference optimization (DPO) is proposed to address those challenges. However, DPO relies on contrastive responses generated from human annotator and alternative LLM, instead of the policy model, limiting the effectiveness of the RLHF. In this paper, we addresses both challenges by systematically combining rejection sampling (RS) and DPO. Our proposed method, RS-DPO, initiates with the development of a supervised fine-tuned policy model (SFT). A varied set of k responses per prompt are sampled directly from the SFT model. RS-DPO identifies pairs of contrastive samples based on their reward distribution. Finally, we apply DPO with the contrastive samples to align the model to human preference. Our experiments indicate that our proposed method effectively fine-tunes LLMs with limited resource environments, leading to improved alignment with user intent. Furthermore, it outperforms existing methods, including RS, PPO, and DPO.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 34 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

25 of 34 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2020Learning to summarize from human feedbackcited by 2,562
9 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  6. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  7. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  9. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 26; 2026: 10.

48 papers.

Sort:

By area:

  1. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  4. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  5. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  6. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  7. 2025A Survey of Direct Preference Optimizationcited by 35
  8. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  9. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  10. 2024The Crucial Role of Samplers in Online Direct Preference Optimizationcited by 26
  11. 2025CasaGPT: Cuboid Arrangement and Scene Assembly for Interior Designcited by 14
  12. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
36 more
  1. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  2. 2024Right Now, Wrong Then: Non-Stationary Direct Preference Optimization under Preference Driftcited by 12
  3. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  4. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
  5. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Processcited by 8
  6. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Datasetcited by 8
  7. 2024Systematic Reward Gap Optimization for Mitigating VLM Hallucinationscited by 7
  8. 2025OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignmentcited by 7
  9. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7
  10. 2024Sparse Rewards Can Self-Train Dialogue Agentscited by 6
  11. 2025Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Rankingcited by 5
  12. 2025Learning a Pessimistic Reward Model in RLHFcited by 5
  13. 2025Scaling-up Perceptual Video Quality Assessmentcited by 5
  14. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  15. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  16. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  17. 2025RePO: Understanding Preference Learning Through ReLU-Based Optimizationcited by 4
  18. 2025Personalized Query Auto-Completion for Long and Short-Term Interests with Adaptive Detoxification Generationcited by 4
  19. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  20. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontiercited by 3
  21. 2025Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomycited by 3
  22. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimizationcited by 3
  23. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  24. 2024Self-Evolution Fine-Tuning for Policy Optimizationcited by 2
  25. 2025Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capablescited by 2
  26. 2026VisTIRA: Closing the Image-Text Modality Gap in Visual Math Reasoning via Structured Tool Integrationcited by 2
  27. 2026Ada-RS: Adaptive Rejection Sampling for Selective Thinkingcited by 1
  28. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  29. 2026Alignment Tuning for Large Language Models: A Data-Centric Lens on Alignment Data Pipelinescited by 1
  30. 2025MetaLint: Easy-to-Hard Generalization for Code Lintingcited by 0
  31. 2025Factors That Support Grounded Responses in LLM Conversations: A Rapid Reviewcited by 0
  32. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Modelscited by 0
  33. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principlescited by 0
  34. 2026The Quality-Utility Paradox: Why High-Reward Data Impairs Small Model Mathematical Reasoningcited by 0
  35. 2026Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leafletcited by 0
  36. 2026FiMI Banking: A Sovereign Model for Indian Retail Bankingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Efficient Exact Optimization of Language Model Alignment
  2. 2025A Principled Loss Function for Direct Language Model Alignment
  3. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  4. 2024Direct Preference Optimization with an Offset
  5. 2024Minor DPO reject penalty to increase training robustness
  6. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  7. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  8. 2024Bootstrapping Language Models with DPO Implicit Rewards