mlsanityData to 7 Oct 2026

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

arXiv 2406.04412 ↗First posted 6 Jun 2024cs.LG cs.AI cs.CLcited by 27

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences becomes a key component to obtaining state-of-the-art performance, but it yields a huge cost to construct a large human-annotated preference dataset. To tackle this problem, we propose a new framework, Spread Preference Annotation with direct preference judgment (SPA), that boosts the alignment of LLMs using only a very small amount of human-annotated preference data. Our key idea is leveraging the human prior knowledge within the small (seed) data and progressively improving the alignment of LLM, by iteratively generating the responses and learning from them with the self-annotated preference data. To be specific, we propose to derive the preference label from the logits of LLM to explicitly extract the model's inherent preference. Compared to the previous approaches using external reward models or implicit in-context learning, we observe that the proposed approach is significantly more effective. In addition, we introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data. Our experimental results demonstrate that the proposed framework significantly boosts the alignment of LLMs. For example, we achieve superior alignment performance on AlpacaEval 2.0 with only 3.3% of the ground-truth preference labels in the Ultrafeedback data compared to the cases using the entire data or state-of-the-art baselines.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 55 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

43 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2024Decoding-time Realignment of Language Models5 Feb
  4. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning4 Dec
  6. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  7. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  8. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2020Learning to summarize from human feedbackcited by 2,562
22 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,053
  5. 2019When Does Label Smoothing Help?cited by 985
  6. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  7. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 691
  12. 2024Self-Rewarding Language Modelscited by 654
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  14. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  15. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  16. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  17. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  18. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  19. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  20. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  21. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  22. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 5.

27 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  3. 2025A Survey of Direct Preference Optimizationcited by 35
  4. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  5. 2025Token-Importance Guided Direct Preference Optimizationcited by 14
  6. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 12
  7. 2025Personalized LLM Decoding via Contrasting Personal Preferencecited by 12
  8. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 9
  9. 2025Preference Optimization by Estimating the Ratio of the Data Distributioncited by 9
  10. 2025RefineCoder: Iterative Improving of Large Language Models via Adaptive Critique Refinement for Code Generationcited by 8
  11. 2024Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimizationcited by 6
  12. 2024SeRA: Self-Reviewing and Alignment of Large Language Models using Implicit Reward Marginscited by 5
15 more
  1. 2025FaST: Feature-aware Sampling and Tuning for Personalized Preference Alignment with Limited Datacited by 5
  2. 2025Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventionscited by 3
  3. 2025Semi-Supervised Preference Optimization with Limited Feedbackcited by 3
  4. 2025SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimizationcited by 3
  5. 2026SAIL: Self-Amplified Iterative Learning for Diffusion Model Alignment with Minimal Human Feedbackcited by 3
  6. 2025OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generationcited by 1
  7. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?cited by 1
  8. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  9. 2025ER-RAG: Enhance RAG with ER-Based Unified Modeling of Heterogeneous Data Sourcescited by 0
  10. 2025Towards Integrated Alignmentcited by 0
  11. 2025Offline Clustering of Preference Learning with Active-data Augmentationcited by 0
  12. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferencescited by 0
  13. 2026S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHFcited by 0
  14. 2026Instruction Quality Matters: Refining Instructions for Effective Preference Learningcited by 0
  15. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  4. 2025Leveraging Robust Optimization for LLM Alignment under Distribution Shifts
  5. 2025A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models
  6. 2024REAL: Response Embedding-based Alignment for LLMs
  7. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  8. 2026Nearly Optimal Active Preference Learning and Its Application to LLM Alignment