# Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

arXiv: https://arxiv.org/abs/2406.04412 · HTML: https://mlsanity.com/paper/2406.04412/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 43)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations](https://mlsanity.com/paper/2312.08935/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback](https://mlsanity.com/paper/2305.14387/) (2023)
- [DivideMix: Learning with Noisy Labels as Semi-supervised Learning](https://mlsanity.com/paper/2002.07394/) (2020)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [ORPO: Monolithic Preference Optimization without Reference Model](https://mlsanity.com/paper/2403.07691/) (2024)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)

## Built on by (most cited first, 27 of 27)

- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Self-Boosting Large Language Models with Synthetic Preference Data](https://mlsanity.com/paper/2410.06961/) (2024)
- [A Survey of Direct Preference Optimization](https://mlsanity.com/paper/2503.11701/) (2025)
- [Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs](https://mlsanity.com/paper/2502.12982/) (2025)
- [Token-Importance Guided Direct Preference Optimization](https://mlsanity.com/paper/2505.19653/) (2025)
- [Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generation](https://mlsanity.com/paper/2502.02088/) (2025)
- [Personalized LLM Decoding via Contrasting Personal Preference](https://mlsanity.com/paper/2506.12109/) (2025)
- [Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment](https://mlsanity.com/paper/2502.18099/) (2025)
- [Preference Optimization by Estimating the Ratio of the Data Distribution](https://mlsanity.com/paper/2505.19601/) (2025)
- [RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generation](https://mlsanity.com/paper/2502.09183/) (2025)
- [Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization](https://mlsanity.com/paper/2409.11212/) (2024)
- [FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Data](https://mlsanity.com/paper/2508.04698/) (2025)
- [SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Margins](https://mlsanity.com/paper/2410.09362/) (2024)
- [Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions](https://mlsanity.com/paper/2502.08657/) (2025)
- [Semi-Supervised Preference Optimization with Limited Feedback](https://mlsanity.com/paper/2511.00040/) (2025)
- [SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization](https://mlsanity.com/paper/2511.06222/) (2025)
- [SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedback](https://mlsanity.com/paper/2602.05380/) (2026)
- [OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generation](https://mlsanity.com/paper/2506.02015/) (2025)
- [Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?](https://mlsanity.com/paper/2508.10530/) (2025)
- [Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning](https://mlsanity.com/paper/2509.19893/) (2025)
- [ER-RAG: Enhance RAG with ER-Based Unified Modeling of Heterogeneous Data Sources](https://mlsanity.com/paper/2504.06271/) (2025)
- [Towards Integrated Alignment](https://mlsanity.com/paper/2508.06592/) (2025)
- [Offline Clustering of Preference Learning with Active-data Augmentation](https://mlsanity.com/paper/2510.26301/) (2025)
- [VRM: Teaching Reward Models to Understand Authentic Human Preferences](https://mlsanity.com/paper/2603.04974/) (2026)
- [S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF](https://mlsanity.com/paper/2607.18258/) (2026)
- [Instruction Quality Matters: Refining Instructions for Effective Preference Learning](https://mlsanity.com/paper/2608.26779/) (2026)
- [AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection](https://mlsanity.com/paper/2609.05899/) (2026)

## Abstract (arXiv)

Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.