mlsanityData to 7 Oct 2026

Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

arXiv 2501.15109 ↗First posted 25 Jan 2025cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on training data quality. In particular, clear quality differences between preferred and rejected responses enhance learning performance. Current methods for identifying and obtaining such high-quality samples demand additional resources or external models. We discover that reference model probability space naturally detects high-quality training samples. Using this insight, we present a sampling strategy that achieves consistent improvements (+0.1 to +0.4) on MT-Bench while using less than half (30-50%) of the training data. We observe substantial improvements (+0.4 to +0.98) for technical tasks (coding, math, and reasoning) across multiple models and hyperparameter settings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 22 references that are in the map. First posted on arXiv 25 Jan 2025.

Builds on

13 of 22 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024Impact of Preference Noise on the Alignment Performance of Generative Language Models15 Apr
  3. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  4. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 8

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuningcited by 28
  7. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
  8. 2024Towards Comprehensive Preference Data Collection for Reward Modelingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  2. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  3. 2024Preference-Guided Reflective Sampling for Aligning Language Models
  4. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets
  5. 2024Filtered Direct Preference Optimization
  6. 2026Mitigating Mismatch within Reference-based Preference Optimization
  7. 2024New Desiderata for Direct Preference Optimization
  8. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback