mlsanityData to 7 Oct 2026

Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration

arXiv 2410.08102 ↗First posted 10 Oct 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Efficient data selection is crucial to accelerate the pretraining of language model (LMs). While various methods have been proposed to enhance data efficiency, limited research has addressed the inherent conflicts between these approaches to achieve optimal data selection for LM pretraining. To tackle this problem, we propose a multi-actor collaborative data selection mechanism: each data selection method independently prioritizes data based on its criterion and updates its prioritization rules using the current state of the model, functioning as an independent actor for data selection; and a console is designed to adjust the impacts of different actors at various stages and dynamically integrate information from all actors throughout the LM pretraining process. We conduct extensive empirical studies to evaluate our multi-actor framework. The experimental results demonstrate that our approach significantly improves data efficiency, accelerates convergence in LM pretraining, and achieves an average relative performance gain up to 10.5% across multiple language model benchmarks compared to the state-of-the-art methods.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

37 of 54 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
25 more
  1. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  2. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  3. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  7. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  8. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  9. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  10. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  11. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  12. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  13. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  14. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  15. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  16. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  17. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 153
  18. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  19. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  20. 2023DoGE: Domain Reweighting with Generalization Estimationcited by 87
  21. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  22. 2017Neural Optimizer Search with Reinforcement Learningcited by 82
  23. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  24. 2024Improving Pretraining Data Using Perplexity Correlationscited by 46
  25. 2024Unlearning Traces the Influential Training Data of Language Modelscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 3.

10 papers.

  1. 2024LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Modelscited by 51
  2. 2025QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretrainingcited by 16
  3. 2026LRAgent: Efficient KV Cache Sharing for Multi-LoRA LLM Agentscited by 10
  4. 2025Selective Learning for Deep Time Series Forecastingcited by 8
  5. 2025ALinFiK: Learning to Approximate Linearized Future Influence Kernel for Scalable Third-Party LLM Data Valuationcited by 6
  6. 2025FactGuard: Leveraging Multi-Agent Systems to Generate Answerable and Unanswerable Questions for Enhanced Long-Context LLM Extractioncited by 1
  7. 2025Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selectioncited by 1
  8. 2025Improving Romanian LLM Pretraining Data using Diversity and Quality Filteringcited by 0
  9. 2026AMO: Operator-level Adaptive Muon Orthogonalizationcited by 0
  10. 2026D3: Dynamic Directional Graph-Constrained Data Scheduling for LLM Trainingcited by 0