mlsanityData to 7 Oct 2026

SMART: Submodular Data Mixture Strategy for Instruction Tuning

arXiv 2403.08370 ↗First posted 13 Mar 2024cs.CL cs.AI cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Instruction Tuning involves finetuning a language model on a collection of instruction-formatted datasets in order to enhance the generalizability of the model to unseen tasks. Studies have shown the importance of balancing different task proportions during finetuning, but finding the right balance remains challenging. Unfortunately, there's currently no systematic method beyond manual tuning or relying on practitioners' intuition. In this paper, we introduce SMART (Submodular data Mixture strAtegy for instRuction Tuning) - a novel data mixture strategy which makes use of a submodular function to assign importance scores to tasks which are then used to determine the mixture weights. Given a fine-tuning budget, SMART redistributes the budget among tasks and selects non-redundant samples from each task. Experimental results demonstrate that SMART significantly outperforms traditional methods such as examples proportional mixing and equal mixing. Furthermore, SMART facilitates the creation of data mixtures based on a few representative subsets of tasks alone and through task pruning analysis, we reveal that in a limited budget setting, allocating budget among a subset of representative tasks yields superior performance compared to distributing the budget among all tasks. The code for reproducing our results is open-sourced at https://github.com/kowndinya-renduchintala/SMART.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 43 of its 79 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

43 of 79 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
31 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,080
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  5. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  6. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  7. 2023The Falcon Series of Open Language Modelscited by 526
  8. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  9. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  10. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  11. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  12. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  13. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  14. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  15. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  16. 2021SIMILAR: Submodular Information Measures Based Active Learning In Realistic Scenarioscited by 83
  17. 2019Learning From Less Data: A Unified Data Subset Selection and Active Learning Framework for Computer Visioncited by 64
  18. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 56
  19. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  20. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  21. 2015An Exploration of Softmax Alternatives Belonging to the Spherical Loss Familycited by 27
  22. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  23. 2022AUTOMATA: Gradient Based Data Subset Selection for Compute-Efficient Hyper-parameter Tuningcited by 22
  24. 2023NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Taskscited by 19
  25. 2023Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Explorationcited by 18
  26. 2022BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruningcited by 17
  27. 2023Becoming self-instruct: introducing early stopping criteria for minimal instruct tuningcited by 13
  28. 2023Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Modelscited by 6
  29. 2021DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptationcited by 5
  30. 2023INGENIOUS: Using Informative Data Subsets for Efficient Pre-Training of Language Modelscited by 2
  31. 2022Partitioned Gradient Matching-based Data Subset Selection for Compute-Efficient Robust ASR Trainingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

  1. 2025Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Datacited by 13
  2. 2025Data Mixing Optimization for Supervised Fine-Tuning of Large Language Modelscited by 12
  3. 2024DEM: Distribution Edited Model for Training with Mixed Data Distributionscited by 3
  4. 2025PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recoverycited by 3
  5. 2025MergeIT: From Selection to Merging for Efficient Instruction Tuningcited by 3
  6. 2025Neural Networks for Learnable and Scalable Influence Estimation of Instruction Fine-Tuning Datacited by 2
  7. 2025Learning Task Mixtures from Task Affinities: A Probabilistic Graphical Model for Supervised Fine-Tuningcited by 1
  8. 2025Subset Selection for Fine-Tuning: A Utility-Diversity Balanced Approach for Mathematical Domain Adaptationcited by 0
  9. 2025Diffusion Reconstruction-based Data Likelihood Estimation for Core-Set Selectioncited by 0
  10. 2026Accelerated Relax-and-Round for Concave Coverage Problemscited by 0
  11. 2026Minibatch Selection for Language Models via Partition Matroid Constrained Gradient Matchingcited by 0
  12. 2026CausalMix: Data Mixture as Causal Inference for Language Model Trainingcited by 0
4 more
  1. 2026Decoupling Generation and Selection for Budget-Constrained Faithful Summarizationcited by 0
  2. 2026Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisationcited by 0
  3. 2026CALIBUDGET: Calibration-Guided Source Allocation for Fixed-Budget Mixed-Reasoning Adaptationcited by 0
  4. 2026A helps B while B hurts A: directed transfer in instruction-tuning mixturecited by 0