mlsanityData to 14 Oct 2026

A Survey on Data Selection for Language Models

arXiv 2402.16827 ↗First posted 26 Feb 2024cs.CL cs.LGcited by 236

Not on the map. Why?

Cited by 25 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research.

Key facts. Cited by 236 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Feb 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 70; 2025: 106; 2026: 60.

236 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  3. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  4. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  5. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  6. 2024Generative Reward Modelscited by 148
  7. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  8. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  9. 2024Language models scale reliably with over-training and on downstream taskscited by 113
  10. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  11. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  12. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
38 more
  1. 2024Scaling Laws for Precisioncited by 95
  2. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  3. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
  4. 2024Instruction Pre-Training: Language Models are Supervised Multitask Learnerscited by 63
  5. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 56
  6. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 52
  7. 2024Adaptive Data Optimization: Dynamic Sample Selection with Scaling Lawscited by 47
  8. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMscited by 46
  9. 2024Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verificationcited by 45
  10. 2024Entropy Law: The Story Behind Data Compression and LLM Performancecited by 44
  11. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  12. 2024SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domaincited by 39
  13. 2024Data Selection via Optimal Control for Language Modelscited by 38
  14. 2024Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplarscited by 37
  15. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  16. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 37
  17. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 35
  18. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 34
  19. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
  20. 2024MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Modelscited by 34
  21. 2024Aioli: A Unified Optimization Framework for Language Model Data Mixingcited by 34
  22. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  23. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  24. 2024Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Modelscited by 29
  25. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 26
  26. 2024How to Synthesize Text Data without Model Collapse?cited by 25
  27. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  28. 2025Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilitiescited by 23
  29. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 23
  30. 2025ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilingualitycited by 22
  31. 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generationcited by 21
  32. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 20
  33. 2024BPO: Towards Balanced Preference Optimization between Knowledge Breadth and Depth in Alignmentcited by 20
  34. 2024The Unmet Promise of Synthetic Training Images: Using Retrieved Real Images Performs Bettercited by 19
  35. 2025Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Modelscited by 19
  36. 2025Language Models Improve When Pretraining Data Matches Target Taskscited by 19
  37. 2024From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Datacited by 18
  38. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18

50 most cited of 236 All 236 (JSON)