mlsanityData to 7 Oct 2026

Rethinking Data Selection at Scale: Random Selection is Almost All You Need

arXiv 2410.09335 ↗First posted 12 Oct 2024cs.CL cs.AIcited by 22

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Supervised fine-tuning (SFT) is crucial for aligning Large Language Models (LLMs) with human instructions. The primary goal during SFT is to select a small yet representative subset of training data from the larger pool, such that fine-tuning with this subset achieves results comparable to or even exceeding those obtained using the entire dataset. However, most existing data selection techniques are designed for small-scale data pools, which fail to meet the demands of real-world SFT scenarios. In this paper, we replicated several self-scoring methods those that do not rely on external model assistance on two million scale datasets, and found that nearly all methods struggled to significantly outperform random selection when dealing with such large-scale data pools. Moreover, our comparisons suggest that, during SFT, diversity in data selection is more critical than simply focusing on high quality data. We also analyzed the limitations of several current approaches, explaining why they perform poorly on large-scale datasets and why they are unsuitable for such contexts. Finally, we found that filtering data by token length offers a stable and efficient method for improving results. This approach, particularly when training on long text data, proves highly beneficial for relatively weaker base models, such as Llama3.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 47 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

24 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Rethinking Data Selection for Supervised Fine-Tuning8 Feb

Other references · 23

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  9. 2023Instruction Tuning with GPT-4cited by 676
  10. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  11. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  12. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
11 more
  1. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  2. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52
  3. 2023Dataset Quantizationcited by 47
  4. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  5. 2023One-Shot Learning as Instruction Data Prospector for Large Language Modelscited by 43
  6. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 42
  7. 2024Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Modelscited by 35
  8. 2024SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflectioncited by 34
  9. 2023Aligning Large Language Models with Human: A Surveycited by 31
  10. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  11. 2023AlpaGasus: Training A Better Alpaca with Fewer Datacited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 9.

22 papers.

Sort:

By area:

  1. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  2. 2025Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencodercited by 11
  3. 2025Winning the Pruning Gamble: A Unified Approach to Joint Sample and Token Pruning for Efficient Supervised Fine-Tuningcited by 8
  4. 2025DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Modelscited by 5
  5. 2026SPICE: Submodular Penalized Information-Conflict Selection for Efficient Large Language Model Trainingcited by 5
  6. 2025Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activationscited by 4
  7. 2025CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMscited by 3
  8. 2026Time Series Reasoning via Process-Verifiable Thinking Data Synthesis and Scheduling for Tailored LLM Reasoningcited by 3
  9. 2025Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMscited by 2
  10. 2025Class-Proportional Coreset Selection for Difficulty-Separable Datacited by 2
  11. 2025Computational Budget Should Be Considered in Data Selectioncited by 2
  12. 2025Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generationcited by 2
10 more
  1. 2025Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuningcited by 1
  2. 2025Data Metabolism: An Efficient Data Design Schema For Vision Language Modelcited by 1
  3. 2026On Representation Redundancy in Large-Scale Instruction Tuning Data Selectioncited by 1
  4. 2026Form Over Content In Gradient-Based Data Attribution Methodscited by 1
  5. 2025Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Setcited by 0
  6. 2026Omnimodal Dataset Distillation via High-order Proxy Alignmentcited by 0
  7. 2026Data Difficulty and the Generalization--Extrapolation Tradeoff in LLM Fine-Tuningcited by 0
  8. 2026Thinking in a Low-Resource Language: What SFT Builds, What RL Fixes, What Accuracy Cannot Seecited by 0
  9. 2026Refreshing Less, Selecting Better: Reusing Stale Gradient Features for Efficient Influence-Based Data Selectioncited by 0
  10. 2026PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Take the essence and discard the dross: A Rethinking on Data Selection for Fine-Tuning Large Language Models
  2. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  3. 2026Diversity in Large Language Models under Supervised Fine-Tuning
  4. 2025Selective Self-to-Supervised Fine-Tuning for Generalization in Large Language Models
  5. 2024Semi-supervised Fine-tuning for Large Language Models
  6. 2026GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection
  7. 2025TACOS: Open Tagging and Comparative Scoring for Instruction Fine-Tuning Data Selection
  8. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data