mlsanityData to 7 Oct 2026

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

arXiv 2409.09603 ↗First posted 15 Sep 2024cs.AI cs.CL cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The goal of aligning language models to human preferences requires data that reveal these preferences. Ideally, time and money can be spent carefully collecting and tailoring bespoke preference data to each downstream application. However, in practice, a select few publicly available preference datasets are often used to train reward models for reinforcement learning from human feedback (RLHF). While new preference datasets are being introduced with increasing frequency, there are currently no existing efforts to measure and compare these datasets. In this paper, we systematically study preference datasets through three perspectives: scale, label noise, and information content. We propose specific metrics for each of these perspectives and uncover different axes of comparison for a better understanding of preference datasets. Our work is a first step towards a data-centric approach to alignment by providing perspectives that aid in training efficiency and iterative data collection for RLHF.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 38 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

36 of 38 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  2. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  5. 2023A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift21 Nov
  6. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  7. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  8. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  10. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
14 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  2. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  3. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  5. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  6. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  7. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  8. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  9. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 201
  10. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  11. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  12. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  13. 2022A Unifying Theory of Distance from Calibrationcited by 51
  14. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 13.

20 papers.

Sort:

By area:

  1. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  2. 2024Optimal Design for Reward Modeling in RLHFcited by 23
  3. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  4. 2025Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's Own Signalscited by 13
  5. 2026Can We Predict Before Executing Machine Learning Agents?cited by 10
  6. 2026Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewardscited by 10
  7. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  8. 2024Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guidecited by 4
  9. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  10. 2026AI Alignment Breaks at the Edgecited by 3
  11. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Modelscited by 3
  12. 2026What Does Preference Learning Recover from Pairwise Comparison Data?cited by 2
8 more
  1. 2026Labels have Human Values: Value Calibration of Subjective Taskscited by 1
  2. 2026CompassDPO: Dynamics-Controlled Direct Preference Optimization for Robust Safety Alignmentcited by 1
  3. 2026RLHF May Not Reflect Genuine Preferencescited by 1
  4. 2026Scaling Reward Modeling without Human Supervisioncited by 0
  5. 2026Aligning to Illusions: Choice Blindness in Human and AI Feedbackcited by 0
  6. 2026Disentangling Optimization Scale from Preference Scale in DPOcited by 0
  7. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0
  8. 2026One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  2. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  3. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  4. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
  5. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  6. 2024How to Evaluate Reward Models for RLHF
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  8. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance