mlsanityData to 14 Oct 2026

Investigating on RLHF methodology

arXiv 2410.01789 ↗First posted 2 Oct 2024cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In this article, we investigate the alignment of Large Language Models according to human preferences. We discuss the features of training a Preference Model, which simulates human preferences, and the methods and details we found essential for achieving the best results. We also discuss using Reinforcement Learning to fine-tune Large Language Models and describe the challenges we faced and the ways to overcome them. Additionally, we present our experience with the Direct Preference Optimization method, which enables us to align a Large Language Model with human preferences without creating a separate Preference Model. As our contribution, we introduce the approach for collecting a preference dataset through perplexity filtering, which makes the process of creating such a dataset for a specific Language Model much easier and more cost-effective.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 20 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

16 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 14

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  10. 2017Generative Adversarial Networks: An Overviewcited by 590
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  12. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
2 more
  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  2. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  2. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  3. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  4. 2024Accelerated Preference Optimization for Large Language Model Alignment
  5. 2025Online Preference Alignment for Language Models via Count-based Exploration
  6. 2025Doubly Robust Alignment for Large Language Models
  7. 2024Towards Efficient Exact Optimization of Language Model Alignment
  8. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective