# Filtered Direct Preference Optimization

arXiv: https://arxiv.org/abs/2404.13846 · HTML: https://mlsanity.com/paper/2404.13846/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 30 of its 42 references that are in the map.
- First posted on arXiv 22 Apr 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Direct Language Model Alignment from Online AI Feedback](https://mlsanity.com/paper/2402.04792/) (2024)
- [Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive](https://mlsanity.com/paper/2402.13228/) (2024)
- [Reward Model Ensembles Help Mitigate Overoptimization](https://mlsanity.com/paper/2310.02743/) (2023)
- [Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking](https://mlsanity.com/paper/2312.09244/) (2023)
- [Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints](https://mlsanity.com/paper/2309.16240/) (2023)
- [Generalized Preference Optimization: A Unified Approach to Offline Alignment](https://mlsanity.com/paper/2402.05749/) (2024)
- [Direct Preference Optimization with an Offset](https://mlsanity.com/paper/2402.10571/) (2024)
- [LiPO: Listwise Preference Optimization through Learning-to-Rank](https://mlsanity.com/paper/2402.01878/) (2024)
- [Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation](https://mlsanity.com/paper/2403.05171/) (2024)
- [Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model](https://mlsanity.com/paper/2403.19443/) (2024)
- [A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift](https://mlsanity.com/paper/2311.14743/) (2023)

## Built on by (most cited first, 24 of 24)

- [β-DPO: Direct Preference Optimization with Dynamic β](https://mlsanity.com/paper/2407.08639/) (2024)
- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Prompt Optimization with Human Feedback](https://mlsanity.com/paper/2405.17346/) (2024)
- [AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization](https://mlsanity.com/paper/2410.10148/) (2024)
- [Geometric-Averaged Preference Optimization for Soft Preference Labels](https://mlsanity.com/paper/2409.06691/) (2024)
- [Towards a Unified View of Preference Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2409.02795/) (2024)
- [Larger or Smaller Reward Margins to Select Preferences for Alignment?](https://mlsanity.com/paper/2503.01864/) (2025)
- [WEPO: Web Element Preference Optimization for LLM-based Web Navigation](https://mlsanity.com/paper/2412.10742/) (2024)
- [Reward-RAG: Enhancing RAG with Reward Driven Supervision](https://mlsanity.com/paper/2410.03780/) (2024)
- [SAFER: Probing Safety in Reward Models with Sparse Autoencoder](https://mlsanity.com/paper/2507.00665/) (2025)
- [REAL: Response Embedding-based Alignment for LLMs](https://mlsanity.com/paper/2409.17169/) (2024)
- [Towards Understanding Valuable Preference Data for Large Language Model Alignment](https://mlsanity.com/paper/2510.13212/) (2025)
- [CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment](https://mlsanity.com/paper/2503.23777/) (2025)
- [Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models](https://mlsanity.com/paper/2511.12464/) (2025)
- [Understanding the Impact of Sampling Quality in Direct Preference Optimization](https://mlsanity.com/paper/2506.04272/) (2025)
- [From Fragments to Facts: A Curriculum-Driven DPO Approach for Generating Hindi News Veracity Explanations](https://mlsanity.com/paper/2507.05179/) (2025)
- [Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization](https://mlsanity.com/paper/2510.08256/) (2025)
- [ViPO: Visual Preference Optimization at Scale](https://mlsanity.com/paper/2604.24953/) (2026)
- [Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learning](https://mlsanity.com/paper/2510.14459/) (2025)
- [Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles](https://mlsanity.com/paper/2605.30619/) (2026)
- [Alchemist: Towards the Design of Efficient Online Continual Learning System](https://mlsanity.com/paper/2503.01066/) (2025)
- [Reliability-Aware Sexism Detection: Combining DPO with Annotator Agreement and Token-Level Confidence Scoring](https://mlsanity.com/paper/2608.12330/) (2026)
- [Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data](https://mlsanity.com/paper/2609.16532/) (2026)

## Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This paper addresses the issue of text quality within the preference dataset by focusing on direct preference optimization (DPO), an increasingly adopted reward-model-free RLHF method. We confirm that text quality significantly influences the performance of models optimized with DPO more than those optimized with reward-model-based RLHF. Building on this new insight, we propose an extension of DPO, termed filtered direct preference optimization (fDPO). fDPO uses a trained reward model to monitor the quality of texts within the preference dataset during DPO training. Samples of lower quality are discarded based on comparisons with texts generated by the model being optimized, resulting in a more accurate dataset. Experimental results demonstrate that fDPO enhances the final model performance. Our code is available at https://github.com/CyberAgentAILab/filtered-dpo.