mlsanityData to 7 Oct 2026

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models

arXiv 2403.07384 ↗First posted 12 Mar 2024cs.CL cs.AI cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

Despite the effectiveness of data selection for large language models (LLMs) during pretraining and instruction fine-tuning phases, improving data efficiency in supervised fine-tuning (SFT) for specialized domains poses significant challenges due to the complexity of fine-tuning data. To bridge this gap, we introduce an effective and scalable data selection method for SFT, SmallToLarge (S2L), which leverages training trajectories from small models to guide the data selection for larger models. We demonstrate through extensive experiments that S2L significantly improves data efficiency in SFT for mathematical problem-solving, reducing the training data to just 11% of the original MathInstruct dataset (Yue et al., 2023) to match full dataset performance while outperforming state-of-the-art data selection algorithms by an average of 4.7% across 6 in- and out-domain evaluation datasets. Remarkably, selecting only 50K data for SFT, S2L achieves a 32.7% accuracy on the most challenging MATH (Hendrycks et al., 2021) benchmark, improving Phi-2 (Li et al., 2023b) by 16.6%. In clinical text summarization on the MIMIC-III dataset (Johnson et al., 2016), S2L again outperforms training on the full dataset using only 50% of the data. Notably, S2L can perform data selection using a reference model 40x smaller than the target model, proportionally reducing the cost of data selection.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 53 of its 72 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

53 of 72 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  2. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  5. 2022Galactica: A Large Language Model for Sciencecited by 736
  6. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  7. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  8. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  10. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
  13. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
  14. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  15. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  16. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  17. 2023Llemma: An Open Language Model For Mathematicscited by 385
  18. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  19. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 337
  20. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  21. 2023Towards Generalist Biomedical AIcited by 286
  22. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  23. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  24. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  25. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  26. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 110
  27. 2022Training Trajectories of Language Models Across Scalescited by 80
  28. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  29. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 68
  30. 2022Adaptive Second Order Coresets for Data-efficient Machine Learningcited by 60
  31. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  32. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  33. 2023Towards Sustainable Learning: Coresets for Data-efficient Deep Learningcited by 44
  34. 2023Robust Learning with Progressive Data Expansion Against Spurious Correlationcited by 39
  35. 2023Sieve: Multimodal Dataset Pruning Using Image Captioning Modelscited by 33
  36. 2022Not All Poisons are Created Equal: Robust Training against Data Poisoningcited by 29
  37. 2023Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Leastcited by 22
  38. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 22

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 22; 2026: 15.

43 papers.

Sort:

By area:

  1. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  2. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  3. 2025The Best Instruction-Tuning Data are Those That Fitcited by 48
  4. 2024Concept-skill Transferability-based Data Selection for Large Vision-Language Modelscited by 24
  5. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 22
  6. 2025Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learningcited by 19
  7. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  8. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  9. 2025Efficient Data Selection at Scale via Influence Distillationcited by 14
  10. 2025Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuningcited by 14
  11. 2025LEAD: Iterative Data Selection for Efficient LLM Instruction Tuningcited by 13
  12. 2025GRAIT: Gradient-Driven Refusal-Aware Instruction Tuning for Effective Hallucination Mitigationcited by 11
31 more
  1. 2024Speculative Coreset Selection for Task-Specific Fine-tuningcited by 10
  2. 2025Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessonscited by 9
  3. 2025Synthetic Text Generation for Training Large Language Models via Gradient Matchingcited by 9
  4. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 9
  5. 2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
  6. 2024HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimationcited by 7
  7. 2025Select2Reason: Efficient Instruction-Tuning Data Selection for Long-CoT Reasoningcited by 7
  8. 2025Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectoriescited by 7
  9. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 5
  10. 2025Swift Cross-Dataset Pruning: Enhancing Fine-Tuning Efficiency in Natural Language Understandingcited by 4
  11. 2025Disentangling the Roles of Representation and Selection in Data Pruningcited by 4
  12. 2025TRIM: Token-wise Attention-Derived Saliency for Data-Efficient Instruction Tuningcited by 3
  13. 2026Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Modelscited by 3
  14. 2025Complexity-aware fine-tuningcited by 2
  15. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  16. 2025Improving Task Diversity in Label Efficient Supervised Finetuning of LLMscited by 1
  17. 2025Synergy over Discrepancy: A Partition-Based Approach to Multi-Domain LLM Fine-Tuningcited by 1
  18. 2025Investigating Data Pruning for Pretraining Biological Foundation Models at Scalecited by 1
  19. 2026On Representation Redundancy in Large-Scale Instruction Tuning Data Selectioncited by 1
  20. 2026GRACE: A Dynamic Coreset Selection Framework for Large Language Model Optimizationcited by 1
  21. 2026From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuningcited by 1
  22. 2026Interaction-Aware Influence Functions for Group Attributioncited by 1
  23. 2026Beyond Temperature: Hyperfitting as a Late-Stage Geometric Expansioncited by 1
  24. 2026Capacity-Dependent Effects of Data Selection for Reasoningcited by 1
  25. 2025Subset Selection for Fine-Tuning: A Utility-Diversity Balanced Approach for Mathematical Domain Adaptationcited by 0
  26. 2026MADS: Model-Aware Diverse Core Set Selection for Instruction Tuningcited by 0
  27. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  28. 2026Small Models Scout Bottleneck Order for Large-Model Data Controlcited by 0
  29. 2026Mapping the Concept Landscape: Structural Perception of Global Distributions for Transparent Data Pruningcited by 0
  30. 2026Clustering-Based Balanced Sampling and Allocation with Data Parallelism for High-Performance Fine-Tuningcited by 0
  31. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0