mlsanityData to 7 Oct 2026

Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning

arXiv 2411.02481 ↗First posted 4 Nov 2024cs.CL cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference tuning relies on high-quality human preference data, which is often expensive and time-consuming to gather. In this paper, we introduce Dr.SoW (Density Ratio of Strong over Weak) a cost-effective method that eliminates the reliance for human annotation by leveraging off-the-shelf LLMs for preference data annotation. Dr.SoW uses the log-density ratio between a better-aligned and a less-aligned LLM as a reward signal. We evaluate Dr.SoW across 221 different LLM pairs and empirically find a strong correlation between the performance gap of the paired models and the quality of the reward signal. This insight provides a practical guideline for selecting LLMs for data annotation. Additionally, we introduce an end-to-end pipeline that customizes reward functions based on user query domains. Without fine-tuning, it improves accuracy on domain-specific evaluations. With a pair of Mistral-7B models, Dr.SoW achieves a RewardBench score of 82.6, outperforming the best trained reward functions from same model class and demonstrating competitive performance against SoTA models in Safety (91.0) and Reasoning (88.0) domains. Further, we preference-tune Llama-3-8B-Instruct using data annotated by Dr.SoW. Our approach pushes Llama-3-8B to achieve a 37.4 % (+15.1 %) win rate on ArenaHard and a 40.7 % (+17.8 %) win rate on length-controlled AlpacaEval 2.0.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 43 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

30 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  4. 2024DPO Meets PPO: Reinforced Token Optimization for RLHF29 Apr
  5. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  6. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  11. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  12. 2024Iterative Reasoning Preference Optimizationcited by 244
9 more
  1. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  2. 2023Contrastive Decoding Improves Reasoning in Large Language Modelscited by 71
  3. 2024Self-Taught Evaluatorscited by 70
  4. 2024Bootstrapping Language Models with DPO Implicit Rewardscited by 53
  5. 2024Direct Judgement Preference Optimizationcited by 33
  6. 2024Distributional Preference Alignment of LLMs via Optimal Transportcited by 26
  7. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 26
  8. 2024Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learningcited by 7
  9. 2024In-Context Learning for Extreme Multi-Label Classificationcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Semi-Supervised Preference Optimization with Limited Feedbackcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Users as Annotators: LLM Preference Learning from Comparison Mode
  2. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  3. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  4. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?
  5. 2024SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
  6. 2026Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
  7. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators
  8. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy