mlsanityData to 7 Oct 2026

Annotation-Efficient Language Model Alignment via Diverse and Representative Response Texts

arXiv 2405.13541 ↗First posted 22 May 2024cs.CL cs.AI cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Preference optimization is a standard approach to fine-tuning large language models to align with human preferences. The quantity, diversity, and representativeness of the preference dataset are critical to the effectiveness of preference optimization. However, obtaining a large amount of preference annotations is difficult in many applications. This raises the question of how to use the limited annotation budget to create an effective preference dataset. To this end, we propose Annotation-Efficient Preference Optimization (AEPO). Instead of exhaustively annotating preference over all available response texts, AEPO selects a subset of responses that maximizes diversity and representativeness from the available responses and then annotates preference over the selected ones. In this way, AEPO focuses the annotation budget on labeling preferences over a smaller but informative subset of responses. We evaluate the performance of preference learning using AEPO on three datasets and show that it outperforms the baselines with the same annotation budget. Our code is available at https://github.com/CyberAgentAILab/annotation-efficient-po

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 137 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

70 of 137 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  3. 2024Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment17 Mar
  4. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  5. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  6. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  7. 2023Direct Preference Optimization for Neural Machine Translation with Minimum Bayes Risk Decoding14 Nov
  8. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  9. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  10. 2023LIMA: Less Is More for Alignment18 May
  11. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
2 more
  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 56

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
38 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  5. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  7. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2020Aligning AI With Shared Human Valuescited by 706
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  12. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  13. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  14. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  15. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  16. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  17. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  18. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  19. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  20. 2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
  21. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  22. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  23. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  24. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  25. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  26. 2018Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Studycited by 90
  27. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  28. 2023Human Feedback is not Gold Standardcited by 82
  29. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 63
  30. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  31. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  32. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  33. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  34. 2023It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Riskcited by 49
  35. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  36. 2022BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Samplingcited by 30
  37. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 23
  38. 2024Instruction Diversity Drives Generalization To Unseen Taskscited by 15

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignmentcited by 4
  2. 2025Model Agnostic Preference Optimization for Medical Image Segmentationcited by 1
  3. 2025Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment
  2. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignment
  3. 2025AIR: A Systematic Analysis of Annotations, Instructions, and Response Pairs in Preference Dataset
  4. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignment
  5. 2025ComPO: Preference Alignment via Comparison Oracles
  6. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  7. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  8. 2025Less is More: Improving LLM Alignment via Preference Data Selection