mlsanityData to 14 Oct 2026

MoDS: Model-oriented Data Selection for Instruction Tuning

arXiv 2311.15653 ↗First posted 27 Nov 2023cs.CLcited by 107

Not on the map. Why?

Abstract (arXiv)

Instruction tuning has become the de facto method to equip large language models (LLMs) with the ability of following user instructions. Usually, hundreds of thousands or millions of instruction-following pairs are employed to fine-tune the foundation LLMs. Recently, some studies show that a small number of high-quality instruction data is enough. However, how to select appropriate instruction data for a given LLM is still an open problem. To address this problem, in this paper we present a model-oriented data selection (MoDS) approach, which selects instruction data based on a new criteria considering three aspects: quality, coverage and necessity. First, our approach utilizes a quality evaluation model to filter out the high-quality subset from the original instruction dataset, and then designs an algorithm to further select from the high-quality subset a seed instruction dataset with good coverage. The seed dataset is applied to fine-tune the foundation LLM to obtain an initial instruction-following LLM. Finally, we develop a necessity evaluation model to find out the instruction data which are performed badly in the initial instruction-following LLM and consider them necessary instructions to further improve the LLMs. In this way, we can get a small high-quality, broad-coverage and high-necessity subset from the original instruction datasets. Experimental results show that, the model fine-tuned with 4,000 instruction pairs selected by our approach could perform better than the model fine-tuned with the full original dataset which includes 214k instruction data.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

24 of 35 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2023LIMA: Less Is More for Alignmentcited by 1,100
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
12 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  2. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  3. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 481
  4. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  5. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  6. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  7. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  8. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  9. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
  10. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 232
  11. 2023On the Exploitability of Instruction Tuningcited by 116
  12. 2023Koala: An Index for Quantifying Overlaps with Pre-training Corporacited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 55; 2025: 34; 2026: 16.

107 papers.

Sort:

By area:

  1. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  3. 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
  4. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  5. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  6. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  7. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 77
  8. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  9. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 62
  10. 2024KeyVideoLLM: Towards Large-scale Video Keyframe Selectioncited by 54
  11. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  12. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
38 more
  1. 2024Explaining Length Bias in LLM-Based Preference Evaluationscited by 49
  2. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  3. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  4. 2024Data Selection via Optimal Control for Language Modelscited by 38
  5. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  6. 2025Large-Scale Data Selection for Instruction Tuningcited by 35
  7. 2023Rethinking the Instruction Quality: LIFT is What You Needcited by 34
  8. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  9. 2024G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translationcited by 30
  10. 2024TAGCOS: Task-agnostic Gradient Clustered Coreset Selection for Instruction Tuning Datacited by 30
  11. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  12. 2024I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigmcited by 27
  13. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  14. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  15. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  16. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 23
  17. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignmentcited by 23
  18. 2025DataFlow: An LLM-Driven Framework for Unified Data Preparation and Workflow Automation in the Era of Data-Centric AIcited by 23
  19. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Needcited by 22
  20. 2025D3: Diversity, Difficulty, and Dependability-Aware Data Selection for Sample-Efficient LLM Instruction Tuningcited by 21
  21. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 21
  22. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  23. 2023Data Management For Training Large Language Models: A Surveycited by 17
  24. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  25. 2024DELIFT: Data Efficient Language model Instruction Fine Tuningcited by 14
  26. 2024Exploring the Mystery of Influential Data for Mathematical Reasoningcited by 13
  27. 2024Chasing Random: Instruction Selection Strategies Fail to Generalizecited by 13
  28. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Modelscited by 12
  29. 2024Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixturescited by 12
  30. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 12
  31. 2024FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Onlycited by 11
  32. 2024Synth-Empathy: Towards High-Quality Synthetic Empathy Datacited by 10
  33. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  34. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
  35. 2024Mastering Collaborative Multi-modal Data Selection: A Focus on Informativeness, Uniqueness, and Representativenesscited by 10
  36. 2024Mosaic-IT: Cost-Free Compositional Data Synthesis for Instruction Tuningcited by 9
  37. 2025ToolACE-R: Model-aware Iterative Training and Adaptive Refinement for Tool Learningcited by 9
  38. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9

50 most cited of 107 All 107 (JSON)