mlsanityData to 14 Oct 2026

Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora

arXiv 2408.12799 ↗First posted 23 Aug 2024cs.CL cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Inconsistent annotations in training corpora, particularly within preference learning datasets, pose challenges in developing advanced language models. These inconsistencies often arise from variability among annotators and inherent multi-dimensional nature of the preferences. To address these issues, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them. Our method enhances preference learning by automatically detecting and selecting consistent annotations. We validate the proposed approach through extensive instruction-following tasks, demonstrating performance improvements of up to 33% across various learning algorithms and proxy capabilities. This work offers a straightforward and reliable solution to address preference inconsistencies without relying on heuristics, serving as an initial step toward the development of more advanced preference learning methodologies. Code is available at https://github.com/Self-Curation/ .

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 38 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

25 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing12 Jun
  2. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedback1 Mar
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  4. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2017Deep Bayesian Active Learning with Image Datacited by 819
  12. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
8 more
  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 425
  3. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  4. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  6. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  7. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
  8. 2023Two Failures of Self-Consistency in the Multi-Step Reasoning of LLMscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Lightweight Robust Direct Preference Optimizationcited by 2
  2. 2025Data Selection for LLM Alignment Using Fine-Grained Preferencescited by 1
  3. 2026Instruction Quality Matters: Refining Instructions for Effective Preference Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  3. 2025TaP: A Taxonomy-Guided Framework for Automated and Scalable Preference Data Generation
  4. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
  5. 2024ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions
  6. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
  7. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators
  8. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models