mlsanityData to 7 Oct 2026

Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model

arXiv 2305.12816 ↗First posted 22 May 2023cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Pretrained language models have achieved remarkable success in various natural language processing tasks. However, pretraining has recently shifted toward larger models and larger data, and this has resulted in significant computational and energy costs. In this paper, we propose Influence Subset Selection (ISS) for language model, which explicitly utilizes end-task knowledge to select a tiny subset of the pretraining corpus. Specifically, the ISS selects the samples that will provide the most positive influence on the performance of the end-task. Furthermore, we design a gradient matching based influence estimation method, which can drastically reduce the computation time of influence. With only 0.45% of the data and a three-orders-of-magnitude lower computational cost, ISS outperformed pretrained models (e.g., RoBERTa) on eight datasets covering four domains.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

34 of 55 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  12. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
22 more
  1. 2020Rethinking Attention with Performerscited by 1,584
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  9. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  10. 2015Image-based Recommendations on Styles and Substitutescited by 702
  11. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
  12. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  14. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  15. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  16. 2021GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Trainingcited by 269
  17. 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 168
  18. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 167
  19. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  20. 2020PMI-Masking: Principled masking of correlated spanscited by 54
  21. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  22. 2022MINER: Improving Out-of-Vocabulary Named Entity Recognition from an Information Theoretic Perspectivecited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 3; 2026: 4.

12 papers.

  1. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  2. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  3. 2024Most Influential Subset Selection: Challenges, Promises, and Beyondcited by 31
  4. 2025Group-Level Data Selection for Efficient Pretrainingcited by 12
  5. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learningcited by 10
  6. 2024In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Modelscited by 9
  7. 2025IF-GUIDE: Influence Function-Guided Detoxification of LLMscited by 7
  8. 2024REQUAL-LM: Reliability and Equity through Aggregation in Large Language Modelscited by 1
  9. 2025An Empirical Study of Sample Selection Strategies for Large Language Model Repaircited by 0
  10. 2026Is More Data Worth the Cost? Dataset Scaling Laws in a Tiny Attention-Only Decodercited by 0
  11. 2026SLAP: Stratified Loss-based Pruning for On-Policy Data-Efficient Instruction Tuningcited by 0
  12. 2026Effective Synthetic Data Curation Requires Group-Level Signalscited by 0