mlsanityData to 14 Oct 2026

Filtered Direct Preference Optimization

arXiv 2404.13846 ↗First posted 22 Apr 2024cs.LG cs.AI cs.CLcited by 24

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO), an increasingly adopted reward-model-free RLHF method. We confirm that text quality significantly influences the performance of models optimized with DPO more than those optimized with reward-model-based RLHF. Building on this new insight, we propose an extension of DPO, termed filtered direct preference optimization (fDPO). fDPO uses a trained reward model to monitor the quality of texts within the preference dataset during DPO training. Samples of lower quality are discarded based on comparisons with texts generated by the model being optimized, resulting in a more accurate dataset. Experimental results demonstrate that fDPO enhances the final model performance. Our code is available at https://github.com/CyberAgentAILab/filtered-dpo.

Key facts. Cited by 24 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 42 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

30 of 42 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model28 Mar
  2. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  3. 2024Direct Preference Optimization with an Offset16 Feb
  4. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  5. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  6. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
9 more
  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  3. 2024Self-Rewarding Language Modelscited by 681
  4. 2016Self-critical Sequence Training for Image Captioningcited by 547
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  6. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  7. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  8. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  9. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shiftcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 12; 2026: 4.

24 papers.

Sort:

By area:

  1. 2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
  2. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  3. 2025A Survey of Direct Preference Optimizationcited by 36
  4. 2024Prompt Optimization with Human Feedbackcited by 35
  5. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 29
  6. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  7. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  8. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?cited by 11
  9. 2024WEPO: Web Element Preference Optimization for LLM-based Web Navigationcited by 10
  10. 2024Reward-RAG: Enhancing RAG with Reward Driven Supervisioncited by 7
  11. 2025SAFER: Probing Safety in Reward Models with Sparse Autoencodercited by 6
  12. 2024REAL: Response Embedding-based Alignment for LLMscited by 4
12 more
  1. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  2. 2025CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignmentcited by 3
  3. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Modelscited by 3
  4. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 2
  5. 2025From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanationscited by 2
  6. 2025Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimizationcited by 2
  7. 2026ViPO: Visual Preference Optimization at Scalecited by 2
  8. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  9. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principlescited by 1
  10. 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
  11. 2026Reliability-Aware Sexism Detection: Combining DPO with Annotator Agreement and Token-Level Confidence Scoringcited by 0
  12. 2026Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024New Desiderata for Direct Preference Optimization
  2. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  3. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  4. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  5. 2024Disentangling Length from Quality in Direct Preference Optimization
  6. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison