mlsanityData to 14 Oct 2026

A Survey on Data Selection for LLM Instruction Tuning

arXiv 2402.05123 ↗First posted 4 Feb 2024cs.CLcited by 77

Not on the map. Why?

Abstract (arXiv)

Instruction tuning is a vital step of training large language models (LLMs), so how to enhance the effect of instruction tuning has received increased attention. Existing works indicate that the quality of the dataset is more crucial than the quantity during instruction tuning of LLMs. Therefore, recently a lot of studies focus on exploring the methods of selecting high-quality subset from instruction datasets, aiming to reduce training costs and enhance the instruction-following capabilities of LLMs. This paper presents a comprehensive survey on data selection for LLM instruction tuning. Firstly, we introduce the wildly used instruction datasets. Then, we propose a new taxonomy of the data selection methods and provide a detailed introduction of recent advances, and the evaluation strategies and results of data selection methods are also elaborated in detail. Finally, we emphasize the open challenges and present new frontiers of this task.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 4 Feb 2024.

Builds on

27 of 38 references.

References · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
15 more
  1. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  2. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  3. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  4. 2023Self-Alignment with Instruction Backtranslationcited by 151
  5. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  6. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  7. 2024A Closer Look at the Limitations of Instruction Tuningcited by 77
  8. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  9. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 58
  10. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
  11. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  12. 2023Dataset Quantizationcited by 48
  13. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  14. 2023Rethinking the Instruction Quality: LIFT is What You Needcited by 34
  15. 2023A Preliminary Study of the Intrinsic Relationship between Complexity and Alignmentcited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 25; 2025: 30; 2026: 22.

77 papers.

Sort:

By area:

  1. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  2. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  3. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  4. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Needcited by 22
  5. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  6. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  7. 2026ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selectioncited by 18
  8. 2024A Multi-Expert Large Language Model Architecture for Verilog Code Generationcited by 15
  9. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  10. 2024Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependencycited by 13
  11. 2024Chasing Random: Instruction Selection Strategies Fail to Generalizecited by 13
  12. 2025LLMs can be easily Confused by Instructional Distractionscited by 13
38 more
  1. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Modelscited by 12
  2. 2024Scaling Parameter-Constrained Language Models with Quality Datacited by 12
  3. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 12
  4. 2025Contrastive Private Data Synthesis via Weighted Multi-PLM Fusioncited by 11
  5. 2024Federated Data-Efficient Instruction Tuning for Large Language Modelscited by 10
  6. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
  7. 2025Active Learning for Direct Preference Optimizationcited by 10
  8. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  9. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  10. 2024LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMscited by 8
  11. 2025LREF: A Novel LLM-based Relevance Framework for E-commercecited by 8
  12. 2026GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometrycited by 8
  13. 2025ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuationcited by 7
  14. 2025Importance-Aware Data Selection for Efficient LLM Instruction Tuningcited by 7
  15. 2025CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdomcited by 6
  16. 2024Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Searchcited by 5
  17. 2025Distributional Machine Unlearning via Selective Data Removalcited by 5
  18. 2025On Code-Induced Reasoning in LLMscited by 5
  19. 2024Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularitycited by 4
  20. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 4
  21. 2025Less is More: Improving LLM Alignment via Preference Data Selectioncited by 4
  22. 2025A Survey of LLM × DATAcited by 4
  23. 2025ArgInstruct: Specialized Instruction Fine-Tuning for Computational Argumentationcited by 4
  24. 2026ReMiT: RL-Guided Mid-Training for Iterative LLM Evolutioncited by 4
  25. 2024Fine-tuning with HED-IT: The impact of human post-editing for dialogical language modelscited by 3
  26. 2024SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipecited by 3
  27. 20243DS: Medical Domain Adaptation of LLMs via Decomposed Difficulty-based Data Selectioncited by 3
  28. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  29. 2025ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuningcited by 3
  30. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  31. 2026Instruction Data Selection via Answer Divergencecited by 3
  32. 2025Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Modelscited by 2
  33. 2025Unveiling Instruction-Specific Neurons & Experts: An Analytical Framework for LLM's Instruction-Following Capabilitiescited by 2
  34. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2
  35. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  36. 2026Finding the Translation Switch: Discovering and Exploiting the Task-Initiation Features in LLMscited by 2
  37. 2026DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learningcited by 2
  38. 2026QiMeng-CodeV-SVA: Training Specialized LLMs for Hardware Assertion Generation via RTL-Grounded Bidirectional Data Synthesiscited by 2

50 most cited of 77 All 77 (JSON)