mlsanityData to 14 Oct 2026

Group-Level Data Selection for Efficient Pretraining

arXiv 2502.14709 ↗First posted 20 Feb 2025cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce Group-MATES, an efficient group-level data selection approach to optimize the speed-quality frontier of language model pretraining. Specifically, Group-MATES parameterizes costly group-level selection with a relational data influence model. To train this model, we sample training trajectories of the language model and collect oracle data influences alongside. The relational data influence model approximates the oracle data influence by weighting individual influence with relationships among training data. To enable efficient selection with our relational data influence model, we partition the dataset into small clusters using relationship weights and select data within each cluster independently. Experiments on DCLM 400M-4x, 1B-1x, and 3B-1x show that Group-MATES achieves 3.5%-9.4% relative performance gains over random selection across 22 downstream tasks, nearly doubling the improvements achieved by state-of-the-art individual data selection baselines. Furthermore, Group-MATES reduces the number of tokens required to reach a certain downstream performance by up to 1.75x, substantially elevating the speed-quality frontier. Further analyses highlight the critical role of relationship weights in the relational data influence model and the effectiveness of our cluster-based inference. Our code is open-sourced at https://github.com/facebookresearch/Group-MATES.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

42 of 53 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  7. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  10. 2024Phi-4 Technical Reportcited by 711
  11. 2019Data Shapley: Equitable Valuation of Data for Machine Learningcited by 629
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
30 more
  1. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  2. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  3. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  4. 2024A Survey on Data Selection for Language Modelscited by 236
  5. 2022Datamodels: Predicting Predictions from Training Datacited by 189
  6. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  7. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  8. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  9. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  10. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  11. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 147
  12. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 133
  13. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 105
  14. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 100
  15. 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 79
  16. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 76
  17. 2024Synthetic continued pretrainingcited by 60
  18. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 53
  19. 2024Data Selection via Optimal Control for Language Modelscited by 38
  20. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  21. 2024Capturing the Temporal Dependence of Training Data Influencecited by 35
  22. 2024Harnessing Diversity for Important Data Selection in Pretraining Large Language Modelscited by 34
  23. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 32
  24. 2020An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?cited by 27
  25. 2022Understanding Instance-Level Impact of Fairness Constraintscited by 27
  26. 2022Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refittingcited by 26
  27. 2022Understanding Influence Functions and Datamodels via Harmonic Analysiscited by 23
  28. 2023Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Modelcited by 12
  29. 2024Approximations to worst-case data dropping: unmasking failure modescited by 8
  30. 2024Generalized Group Data Attributioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

  1. 2026OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iterationcited by 21
  2. 2026Influence-Preserving Proxies for Gradient-Based Data Selection in LLM Fine-tuningcited by 10
  3. 2025RePro: Training Language Models to Faithfully Recycle the Web for Pretrainingcited by 6
  4. 2026Interaction-Aware Influence Functions for Group Attributioncited by 2
  5. 2026Generating Pretraining Tokens from Organic Data for Data-Bound Scalingcited by 2
  6. 2026Hubs or Fringes: Pretraining Data Selection via Web Graph Centralitycited by 2
  7. 2026Domain-Aware Scaling Laws Uncover Data Synergycited by 2
  8. 2026PRISM: Preference-Aware Influence Function Based Data Selection Method for Efficient Fine-Tuningcited by 1
  9. 2026One Algorithm, Two Goals: Dual Scoring for Parameter and Data Selection in LLM Fine-Tuningcited by 0
  10. 2026D3: Dynamic Directional Graph-Constrained Data Scheduling for LLM Trainingcited by 0
  11. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0
  12. 2026What Should Data Teach? Moving Bottlenecks Across Circuit, Store, and Usecited by 0
1 more
  1. 2026Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Modelscited by 0