mlsanityData to 14 Oct 2026

What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

arXiv 2312.15685 ↗First posted 25 Dec 2023cs.CL cs.AI cs.LGcited by 364

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, deita performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples -- over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently.

Key facts. Cited by 364 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 34 references that are in the map. First posted on arXiv 25 Dec 2023.

Builds on

25 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment10 Aug
  2. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision4 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 20

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  11. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
8 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  2. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  3. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  4. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  5. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  6. 2023Self-Alignment with Instruction Backtranslationcited by 151
  7. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  8. 2023TART: A plug-and-play Transformer module for task-agnostic reasoningcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 147; 2025: 117; 2026: 100.

364 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2024Yi: Open Foundation Models by 01.AIcited by 719
  3. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  4. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  6. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  7. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  8. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 158
  9. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  10. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  11. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  12. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
38 more
  1. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  2. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  3. 2024DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solvingcited by 95
  4. 2024Stable LM 2 1.6B Technical Reportcited by 90
  5. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  6. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  7. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  8. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  9. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  10. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  11. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  12. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 62
  13. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 59
  14. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  15. 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 56
  16. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  17. 2025Intern-S1: A Scientific Multimodal Foundation Modelcited by 53
  18. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  19. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  20. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  21. 2024Reinforcement Learning from Human Feedback with Active Queriescited by 43
  22. 2024Discovering Preference Optimization Algorithms with and for Large Language Modelscited by 42
  23. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  24. 2024Instruction Tuning With Loss Over Instructionscited by 40
  25. 2024Memory3: Language Modeling with Explicit Memorycited by 40
  26. 2024Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMscited by 40
  27. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  28. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  29. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  30. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  31. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  32. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  33. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  34. 2024Evaluating Language Models as Synthetic Data Generatorscited by 33
  35. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  36. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  37. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 30
  38. 2024Diversity Measurement and Subset Selection for Instruction Tuning Datasetscited by 29

50 most cited of 364 All 364 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MAIN: Mutual Alignment Is Necessary for instruction tuning
  2. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  3. 2025Aligning Instruction Tuning with Pre-training
  4. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning
  5. 2024Align2LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation
  6. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  7. 2023Rethinking the Instruction Quality: LIFT is What You Need
  8. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning