# A Survey on Data Selection for LLM Instruction Tuning

arXiv: https://arxiv.org/abs/2402.05123 · HTML: https://mlsanity.com/paper/2402.05123/

## Key facts

- Cited by 77 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 4 Feb 2024.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://mlsanity.com/paper/2312.15685/) (2023)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Models](https://mlsanity.com/paper/2308.07074/) (2023)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [A Closer Look at the Limitations of Instruction Tuning](https://mlsanity.com/paper/2402.05119/) (2024)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning](https://mlsanity.com/paper/2305.09246/) (2023)
- [Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks](https://mlsanity.com/paper/2311.00288/) (2023)
- [Self-Evolved Diverse Data Sampling for Efficient Instruction Tuning](https://mlsanity.com/paper/2311.08182/) (2023)
- [Dataset Quantization](https://mlsanity.com/paper/2308.10524/) (2023)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [Rethinking the Instruction Quality: LIFT is What You Need](https://mlsanity.com/paper/2312.11508/) (2023)
- [A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment](https://mlsanity.com/paper/2308.05696/) (2023)

## Built on by (most cited first, 30 of 77)

- [Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Models](https://mlsanity.com/paper/2412.02980/) (2024)
- [Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models](https://mlsanity.com/paper/2408.02085/) (2024)
- [Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric](https://mlsanity.com/paper/2502.17184/) (2025)
- [Rethinking Data Selection at Scale: Random Selection is Almost All You Need](https://mlsanity.com/paper/2410.09335/) (2024)
- [Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation](https://mlsanity.com/paper/2508.10858/) (2025)
- [VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models](https://mlsanity.com/paper/2503.07478/) (2025)
- [ProFit: Leveraging High-Value Signals in SFT via Probability-Guided Token Selection](https://mlsanity.com/paper/2601.09195/) (2026)
- [A Multi-Expert Large Language Model Architecture for Verilog Code Generation](https://mlsanity.com/paper/2404.08029/) (2024)
- [The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective](https://mlsanity.com/paper/2407.08583/) (2024)
- [Beyond IID: Optimizing Instruction Learning from the Perspective of Instruction Interaction and Dependency](https://mlsanity.com/paper/2409.07045/) (2024)
- [Chasing Random: Instruction Selection Strategies Fail to Generalize](https://mlsanity.com/paper/2410.15225/) (2024)
- [LLMs can be easily Confused by Instructional Distractions](https://mlsanity.com/paper/2502.04362/) (2025)
- [Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models](https://mlsanity.com/paper/2406.14115/) (2024)
- [Scaling Parameter-Constrained Language Models with Quality Data](https://mlsanity.com/paper/2410.03083/) (2024)
- [Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder](https://mlsanity.com/paper/2502.14050/) (2025)
- [Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion](https://mlsanity.com/paper/2502.00245/) (2025)
- [Federated Data-Efficient Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2410.10926/) (2024)
- [ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuning](https://mlsanity.com/paper/2412.00631/) (2024)
- [Active Learning for Direct Preference Optimization](https://mlsanity.com/paper/2503.01076/) (2025)
- [Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuning](https://mlsanity.com/paper/2602.17835/) (2026)
- [A Survey on Efficient Large Language Model Training: From Data-centric Perspectives](https://mlsanity.com/paper/2510.25817/) (2025)
- [LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMs](https://mlsanity.com/paper/2408.13467/) (2024)
- [LREF: A Novel LLM-based Relevance Framework for E-commerce](https://mlsanity.com/paper/2503.09223/) (2025)
- [GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometry](https://mlsanity.com/paper/2602.18584/) (2026)
- [ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuation](https://mlsanity.com/paper/2503.01052/) (2025)
- [Importance-Aware Data Selection for Efficient LLM Instruction Tuning](https://mlsanity.com/paper/2511.07074/) (2025)
- [CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdom](https://mlsanity.com/paper/2503.01836/) (2025)
- [Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search](https://mlsanity.com/paper/2410.10392/) (2024)
- [Distributional Machine Unlearning via Selective Data Removal](https://mlsanity.com/paper/2507.15112/) (2025)
- [On Code-Induced Reasoning in LLMs](https://mlsanity.com/paper/2509.21499/) (2025)

## Abstract (arXiv)

Instruction tuning is a vital step of training large language models (LLMs), so how to enhance the effect of instruction tuning has received increased attention. Existing works indicate that the quality of the dataset is more crucial than the quantity during instruction tuning of LLMs. Therefore, recently a lot of studies focus on exploring the methods of selecting high-quality subset from instruction datasets, aiming to reduce training costs and enhance the instruction-following capabilities of LLMs. This paper presents a comprehensive survey on data selection for LLM instruction tuning. Firstly, we introduce the wildly used instruction datasets. Then, we propose a new taxonomy of the data selection methods and provide a detailed introduction of recent advances, and the evaluation strategies and results of data selection methods are also elaborated in detail. Finally, we emphasize the open challenges and present new frontiers of this task.