mlsanityData to 14 Oct 2026

Compute-Constrained Data Selection

arXiv 2410.16208 ↗First posted 21 Oct 2024cs.LG cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Data selection can reduce the amount of training data needed to finetune LLMs; however, the efficacy of data selection scales directly with its compute. Motivated by the practical challenge of compute-constrained finetuning, we consider the setting in which both the cost of selecting data and training are budgeted for. We first formalize the problem of data selection with a cost-aware utility function, and model the data selection problem as trading off initial-selection cost for training gain. We run a comprehensive sweep of experiments across multiple tasks, varying compute budget by scaling finetuning tokens, model sizes, and data selection compute. Interestingly we find that many powerful data selection methods are almost never compute-optimal, and that cheaper data selection alternatives dominate both from a theoretical and empirical perspective. For compute-optimal training, we find that perplexity and gradient data selection require training-to-selection model size ratios of 5x and 10x, respectively.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 21 Oct 2024.

Builds on

42 of 53 references.

References · 42

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  9. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
30 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  7. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  8. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  10. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  11. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  12. 2023Scaling Data-Constrained Language Modelscited by 403
  13. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  14. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  15. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 274
  16. 2021Scaling Laws for Transfercited by 272
  17. 2024A Survey on Data Selection for Language Modelscited by 236
  18. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  19. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 226
  20. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  21. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  22. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 121
  23. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
  24. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 84
  25. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  26. 2024Scaling Laws for Downstream Task Performance of Large Language Modelscited by 49
  27. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  28. 2022Data-Efficient Finetuning Using Cross-Task Nearest Neighborscited by 26
  29. 2023In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuningcited by 19
  30. 2020Selecting Informative Contexts Improves Language Model Finetuningcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2025Efficient Data Selection at Scale via Influence Distillationcited by 15
  2. 2025Influence Functions for Efficient Data Selection in Reasoningcited by 10
  3. 2026GIST: Targeted Data Selection for Instruction Tuning via Coupled Optimization Geometrycited by 8
  4. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Modelscited by 5
  5. 2026A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)cited by 4
  6. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  7. 2026On Representation Redundancy in Large-Scale Instruction Tuning Data Selectioncited by 1
  8. 2026Efficient Data Selection for Multimodal Models via Incremental Optimization Utilitycited by 0
  9. 2026Unifying Data, Memory, and Compute Efficiency in LLM training: A Surveycited by 0
  10. 2026Spokes: Optimizing for Diverse Pretraining Data Selectioncited by 0
  11. 2026Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transportcited by 0
  12. 2026LESSER: Post-Training Data Selection with Output-Layer Gradientscited by 0