mlsanityData to 14 Oct 2026

Alignment Data Map for Efficient Preference Data Selection and Diagnosis

arXiv 2505.23114 ↗First posted 29 May 2025cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Human preference data is essential for aligning large language models (LLMs) with human values, but collecting such data is often costly and inefficient-motivating the need for efficient data selection methods that reduce annotation costs while preserving alignment effectiveness. To address this issue, we propose Alignment Data Map, a data analysis tool for identifying and selecting effective preference data. We first evaluate alignment scores of the preference data by LLM-as-a-judge, explicit reward model, and reference-based approaches. The Alignment Data Map considers both response quality and inter-response variability based on the alignment scores. From our experimental findings, training on only 33% of samples that exhibit high-quality and low-variability, achieves comparable or superior alignment performance on MT-Bench, Evol-Instruct, and AlpacaEval, compared to training with the full dataset. In addition, Alignment Data Map detects potential label misannotations by analyzing correlations between annotated labels and alignment scores, improving annotation accuracy. The implementation is available at https://github.com/01choco/Alignment-Data-Map.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 38 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

28 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs10 Oct
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
11 more
  1. 2023Large Language Modelscited by 780
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  4. 2020Artificial Intelligence, Values and Alignmentcited by 448
  5. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  7. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  8. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  9. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  10. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
  11. 2024Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025CancerGUIDE: Cancer Guideline Understanding via Internal Disagreement Estimationcited by 1
  2. 2026Annotation Entropy Predicts Per-Example Learning Dynamics in LoRA Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment
  2. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  3. 2025Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
  4. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?
  5. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  6. 2025ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2025ComPO: Preference Alignment via Comparison Oracles