mlsanityData to 14 Oct 2026

DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization

arXiv 2506.14157 ↗First posted 17 Jun 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Recent research has attempted to associate preference optimization (PO) performance with the underlying preference datasets. In this work, our observation is that the differences between the preferred response y^+ and dispreferred response y^- influence what LLMs can learn, which may not match the desirable differences to learn. Therefore, we use distance and reward margin to quantify these differences, and combine them to get Distance Calibrated Reward Margin (DCRM), a metric that measures the quality of a response pair for PO. Intuitively, DCRM encourages minimal noisy differences and maximal desired differences. With this, we study 3 types of commonly used preference datasets, classified along two axes: the source of the responses and the preference labeling function. We establish a general correlation between higher DCRM of the training set and better learning outcome. Inspired by this, we propose a best-of-N2 pairing method that selects response pairs with the highest DCRM. Empirically, in various settings, our method produces training datasets that can further improve models' performance on AlpacaEval, MT-Bench, and Arena-Hard over the existing training sets.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 43 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

36 of 43 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Assessing Robustness to Spurious Correlations in Post-Training Language Models9 May
  2. 2024Geometric-Averaged Preference Optimization for Soft Preference Labels10 Sep
  3. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment12 Aug
  4. 2024Robust Preference Optimization through Reward Model Distillation29 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Direct Preference Optimization with an Offset16 Feb
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  10. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment7 Nov
  11. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
2 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  11. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
10 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  4. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  5. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  7. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  10. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
  2. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  3. 2026Token-weighted Direct Preference Optimization with Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  2. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principles
  3. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
  4. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
  5. 2026Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization
  6. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winner
  7. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  8. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback