mlsanityData to 7 Oct 2026

Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment

arXiv 2403.11124 ↗First posted 17 Mar 2024cs.CL cs.AIcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there are two different ways of allocating considered: more diverse PROMPTS or more diverse RESPONSES to be labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence. We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and conduct experiments to show its effect on different algorithms.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

38 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Making Large Language Models Better Reasoners with Alignment5 Sep
  2. 2023Preference Ranking Optimization for Human Alignment30 Jun
  3. 2023Fine-Tuning Language Models with Advantage-Induced Policy Alignment4 Jun
  4. 2023LIMA: Less Is More for Alignment18 May
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
21 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2022Transformers learn in-context by gradient descentcited by 714
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  13. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  14. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  15. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  16. 2023Wider and Deeper LLM Networks are Fairer LLM Evaluatorscited by 91
  17. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  18. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15
  19. 2023How Does In-Context Learning Help Prompt Tuning?cited by 11
  20. 2023Constructive Large Language Models Alignment with Diverse Feedbackcited by 9
  21. 2022Improving Task Generalization via Unified Schema Promptcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 11; 2026: 6.

25 papers.

Sort:

By area:

  1. 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
  2. 2024Self-Boosting Large Language Models with Synthetic Preference Datacited by 40
  3. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  4. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  5. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  6. 2024Packing Analysis: Packing Is More Appropriate for Large Models or Datasets in Supervised Fine-tuningcited by 11
  7. 2025Well Begun is Half Done: Low-resource Preference Alignment by Weak-to-Strong Decodingcited by 7
  8. 2024TODO: Enhancing LLM Alignment with Ternary Preferencescited by 5
  9. 2025Measuring Diversity in Synthetic Datasetscited by 5
  10. 2025A survey of diversity quantification in natural language processing: The why, what, where and howcited by 4
  11. 2024Annotation-Efficient Language Model Alignment via Diverse and Representative Response Textscited by 3
  12. 2025From Macro to Micro: Probing Dataset Diversity in Language Model Fine-Tuningcited by 3
13 more
  1. 2025Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answeringcited by 3
  2. 2024LLM-DER:A Named Entity Recognition Method Based on Large Language Models for Chinese Coal Chemical Domaincited by 2
  3. 2024AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuningcited by 2
  4. 2025Exploring Solution Divergence and Its Effect on Large Language Model Problem Solvingcited by 2
  5. 2026Charting Empirical Laws for LLM Fine-Tuning in Scientific Multi-Discipline Learningcited by 2
  6. 2025P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesiscited by 1
  7. 2026Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversitycited by 1
  8. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandingscited by 0
  9. 2025HAMMER: Hamiltonian Curiosity Augmented Large Language Model Reinforcementcited by 0
  10. 2026Measuring Dataset Diversity from a Geometric Perspectivecited by 0
  11. 2026A Diversity Diet for a Healthier Model: A Case Study of French ModernBERTcited by 0
  12. 2026EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptationcited by 0
  13. 2026Escaping the Mode Lottery: Multi-Response Training Improves Language Model Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On Diversified Preferences of Large Language Model Alignment
  2. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  3. 2025DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values
  4. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  5. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  6. 2024Uncovering Factor Level Preferences to Improve Human-Model Alignment
  7. 2025Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment
  8. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language