mlsanityData to 7 Oct 2026

Automatic Document Selection for Efficient Encoder Pretraining

arXiv 2210.10951 ↗First posted 20 Oct 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Building pretrained language models is considered expensive and data-intensive, but must we increase dataset size to achieve better performance? We propose an alternative to larger training sets by automatically identifying smaller yet domain-representative subsets. We extend Cynical Data Selection, a statistical sentence scoring method that conditions on a representative target domain corpus. As an example, we treat the OntoNotes corpus as a target domain and pretrain a RoBERTa-like encoder from a cynically selected subset of the Pile. On both perplexity and across several downstream tasks in the target domain, it consistently outperforms random selection with 20x less data, 3x fewer training iterations, and 2x less estimated cloud compute cost, validating the recipe of automatic document selection for LM pretraining.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 20 Oct 2022.

Builds on

29 of 44 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  7. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2019How Can We Know What Language Models Know?cited by 953
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
17 more
  1. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  2. 2020GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learningcited by 224
  3. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  4. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  5. 2019Variational Pretraining for Semi-supervised Text Classificationcited by 41
  6. 2020Code and Named Entity Recognition in StackOverflowcited by 39
  7. 2017Dynamic Data Selection for Neural Machine Translationcited by 33
  8. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  9. 2021Pre-train or Annotate? Domain Adaptation with a Constrained Budgetcited by 27
  10. 2021Should We Be Pre-training? An Argument for End-task Aware Training as an Alternativecited by 26
  11. 2020Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Mediacited by 16
  12. 2017Cynical Selection of Language Model Training Datacited by 13
  13. 2021Quantifying Adaptability in Pre-trained Language Models with 500 Taskscited by 13
  14. 2022Towards Computationally Feasible Deep Active Learningcited by 11
  15. 2018Neural-Davidsonian Semantic Proto-role Labelingcited by 9
  16. 2018Learning to Selectively Transfer: Reinforced Transfer Learning for Deep Text Matchingcited by 4
  17. 2021AutoSampling: Search for Effective Data Sampling Schedulescited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4; 2025: 4.

11 papers.

  1. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  2. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  3. 2024DsDm: Model-Aware Dataset Selection with Datamodelscited by 103
  4. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  5. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  6. 2025Efficient Data Selection at Scale via Influence Distillationcited by 14
  7. 2025Merge to Mix: Mixing Datasets via Model Mergingcited by 7
  8. 2024Task Oriented In-Domain Data Augmentationcited by 6
  9. 2025Approximating Language Model Training Data from Weightscited by 4
  10. 2025Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Datacited by 4
  11. 2024Generative Deduplication For Socia Media Data Selectioncited by 2