mlsanityData to 14 Oct 2026

Instruction Mining: Instruction Data Selection for Tuning Large Language Models

arXiv 2307.06290 ↗First posted 12 Jul 2023cs.CL cs.AI cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are initially pretrained for broad capabilities and then finetuned with instruction-following datasets to improve their performance in interacting with humans. Despite advances in finetuning, a standardized guideline for selecting high-quality datasets to optimize this process remains elusive. In this paper, we first propose InstructMining, an innovative method designed for automatically selecting premium instruction-following data for finetuning LLMs. Specifically, InstructMining utilizes natural language indicators as a measure of data quality, applying them to evaluate unseen datasets. During experimentation, we discover that double descent phenomenon exists in large language model finetuning. Based on this observation, we further leverage BlendSearch to help find the best subset among the entire dataset (i.e., 2,532 out of 100,000). Experiment results show that InstructMining-7B achieves state-of-the-art performance on two of the most popular benchmarks: LLM-as-a-judge and Huggingface OpenLLM leaderboard.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

18 of 30 references.

References · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  11. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
6 more
  1. 2023Instruction Tuning with GPT-4cited by 686
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
  3. 2023Textbooks Are All You Needcited by 569
  4. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  5. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  6. 2019FLAML: A Fast and Lightweight AutoML Librarycited by 106

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 19; 2025: 24; 2026: 10.

58 papers.

Sort:

By area:

  1. 2023Efficient Large Language Models: A Surveycited by 186
  2. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  3. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  4. 2024LLM2LLM: Boosting LLMs with Novel Iterative Data Enhancementcited by 70
  5. 2024Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selectioncited by 48
  6. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  7. 2025Token Cleaning: Fine-Grained Data Selection for LLM Supervised Fine-Tuningcited by 39
  8. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  9. 2023Rethinking the Instruction Quality: LIFT is What You Needcited by 34
  10. 2023Aligning Large Language Models with Human: A Surveycited by 32
  11. 2024Improving Data Efficiency via Curating LLM-Driven Rating Systemscited by 29
  12. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
38 more
  1. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  2. 2025FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Trainingcited by 19
  3. 2024Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spacescited by 17
  4. 2025Don't Let It Hallucinate: Premise Verification via Retrieval-Augmented Logical Reasoningcited by 15
  5. 2025Efficient Data Selection at Scale via Influence Distillationcited by 15
  6. 2025R.I.P.: Better Models by Survival of the Fittest Promptscited by 13
  7. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  8. 2025T-SHIRT: Token-Selective Hierarchical Data Selection for Instruction Tuningcited by 13
  9. 2024OPTune: Efficient Online Preference Tuningcited by 10
  10. 2025AutoSpatial: Visual-Language Reasoning for Social Robot Navigation through Efficient Spatial Reasoning Learningcited by 10
  11. 2024Octo-planner: On-device Language Model for Planner-Action Agentscited by 8
  12. 2024HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimationcited by 7
  13. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  14. 2025Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoningcited by 7
  15. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  16. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 7
  17. 2024Baichuan Alignment Technical Reportcited by 6
  18. 2025Towards Active Synthetic Data Generation for Finetuning Language Modelscited by 6
  19. 2025Measuring Diversity in Synthetic Datasetscited by 5
  20. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 4
  21. 2025A Survey of LLM × DATAcited by 4
  22. 2024A Bayesian Approach to Data Point Selectioncited by 3
  23. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  24. 2025On the Role of Preference Variance in Preference Optimizationcited by 3
  25. 2024Small Language Model as Data Prospector for Large Language Modelcited by 2
  26. 2024NILE: Internal Consistency Alignment in Large Language Modelscited by 2
  27. 2025Understanding the Impact of Sampling Quality in Direct Preference Optimizationcited by 2
  28. 2026OrderDP: A Theoretically Guaranteed Lossless Dynamic Data Pruning Frameworkcited by 2
  29. 2023SkyMath: Technical Reportcited by 1
  30. 2024From Symbolic Tasks to Code Generation: Diversification Yields Better Task Performerscited by 1
  31. 2024Boosting LLM via Learning from Data Iteratively and Selectivelycited by 1
  32. 2025Holdout-Loss-Based Data Selection for LLM Finetuning via In-Context Learningcited by 1
  33. 2025Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuningcited by 1
  34. 2026Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learningcited by 1
  35. 2024Prioritizing Informative Features and Examples for Deep Learning from Noisy Datacited by 0
  36. 2024Learning from "Silly" Questions Improves Large Language Models, But Only Slightlycited by 0
  37. 2025Towards Efficient and Effective Alignment of Large Language Modelscited by 0
  38. 2025Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMscited by 0

50 most cited of 58 All 58 (JSON)