mlsanityData to 7 Oct 2026

Chasing Random: Instruction Selection Strategies Fail to Generalize

arXiv 2410.15225 ↗First posted 19 Oct 2024cs.AI cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Prior work has shown that language models can be tuned to follow user instructions using only a small set of high-quality instructions. This has accelerated the development of methods that filter a large, noisy instruction-tuning datasets down to high-quality subset which works just as well. However, typically, the performance of these methods is not demonstrated across a uniform experimental setup and thus their generalization capabilities are not well established. In this work, we analyze popular selection strategies across different source datasets, selection budgets and evaluation benchmarks: Our results indicate that selection strategies generalize poorly, often failing to consistently outperform even random baselines. We also analyze the cost-performance trade-offs of using data selection. Our findings reveal that data selection can often exceed the cost of fine-tuning on the full dataset, yielding only marginal and sometimes no gains compared to tuning on the full dataset or a random subset.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 28 of its 44 references that are in the map. First posted on arXiv 19 Oct 2024.

Builds on

28 of 44 references.

References · 28

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  7. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  9. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  10. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  11. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
16 more
  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2024Are We Done with MMLU?cited by 178
  3. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 134
  4. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  5. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  6. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  7. 2023MM-LIMA: Less Is More for Alignment in Multi-Modal Datasetscited by 61
  8. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  9. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  10. 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35
  11. 2024Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Modelscited by 32
  12. 2024G-DIG: Towards Gradient-based Diverse and High-quality Instruction Data Selection for Machine Translationcited by 29
  13. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  14. 2024Annotator-Centric Active Learning for Subjective NLP Taskscited by 18
  15. 2023Strings from the Library of Babel: Random Sampling as a Strong Baseline for Prompt Optimisationcited by 7
  16. 2024Stronger Random Baselines for In-Context Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 5.

12 papers.

  1. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  2. 2025Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metriccited by 24
  3. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 22
  4. 2025SimBA: Simplifying Benchmark Analysis Using Performance Matrices Alonecited by 5
  5. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  6. 2025Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuningcited by 1
  7. 2025LASER: Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategycited by 1
  8. 2026From Instance Selection to Fixed-Pool Data Recipe Search for Supervised Fine-Tuningcited by 1
  9. 2026BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validitycited by 0
  10. 2026Learning Multi-Indicator Weights for Data Selection: A Joint Task-Model Adaptation Framework with Efficient Proxiescited by 0
  11. 2026C-Instrument: Automating RL Data Generation and Hillclimbing with a Constitution-Grid Instrumentcited by 0
  12. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0