mlsanityData to 7 Oct 2026

NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework

arXiv 2111.04130 ↗First posted 7 Nov 2021cs.CL cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Pretrained language models have become the standard approach for many NLP tasks due to strong performance, but they are very expensive to train. We propose a simple and efficient learning framework, TLM, that does not rely on large-scale pretraining. Given some labeled task data and a large general corpus, TLM uses task data as queries to retrieve a tiny subset of the general corpus and jointly optimizes the task objective and the language modeling objective from scratch. On eight classification datasets in four domains, TLM achieves results better than or similar to pretrained language models (e.g., RoBERTa-Large) while reducing the training FLOPs by two orders of magnitude. With high accuracy and efficiency, we hope TLM will contribute to democratizing NLP and expediting its development.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 41 of its 59 references that are in the map. First posted on arXiv 7 Nov 2021.

Builds on

41 of 59 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
29 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  7. 2019Are Sixteen Heads Really Better than One?cited by 803
  8. 2015Image-based Recommendations on Styles and Substitutescited by 702
  9. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  10. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  11. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  12. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  13. 2019Revealing the Dark Secrets of BERTcited by 300
  14. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  15. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  16. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  17. 2019Structured Pruning of Large Language Modelscited by 201
  18. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  19. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  20. 2018Deep Co-Training for Semi-Supervised Image Recognitioncited by 145
  21. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  22. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  23. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  24. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  25. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  26. 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
  27. 2020Deep Co-Training with Task Decomposition for Semi-Supervised Domain Adaptationcited by 41
  28. 2020Train No Evil: Selective Masking for Task-Guided Pre-Trainingcited by 36
  29. 2020Optimal Subarchitecture Extraction For BERTcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 12; 2023: 14; 2024: 4; 2025: 4.

34 papers.

Sort:

By area:

  1. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  4. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  5. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  6. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  7. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  8. 2024TSDS: Data Selection for Task-Specific Model Finetuningcited by 28
  9. 2023Too Large; Data Reduction for Vision-Language Pre-Trainingcited by 26
  10. 2023The Efficiency Spectrum of Large Language Models: An Algorithmic Surveycited by 25
  11. 2022GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Costcited by 24
  12. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
22 more
  1. 2025Ten Challenging Problems in Federated Foundation Modelscited by 20
  2. 2022Active Multi-Task Representation Learningcited by 13
  3. 2023GIO: Gradient Information Optimization for Training Dataset Selectioncited by 13
  4. 2022AANG: Automating Auxiliary Learningcited by 12
  5. 2023Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Modelcited by 12
  6. 2024Unearthing Large Scale Domain-Specific Knowledge from Public Corporacited by 12
  7. 2022Domain-Specific NER via Retrieving Correlated Samplescited by 11
  8. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 11
  9. 2023Improved Active Multi-Task Representation Learning via Lassocited by 9
  10. 2024Exploring Forgetting in Large Language Model Pre-Trainingcited by 9
  11. 2022Parameter-Efficient Tuning with Special Token Adaptationcited by 7
  12. 2023TCM-GPT: Efficient Pre-training of Large Language Models for Domain Adaptation in Traditional Chinese Medicinecited by 7
  13. 2022Knowledge-grounded Dialog State Trackingcited by 5
  14. 2023Distributional Data Augmentation Methods for Low Resource Languagecited by 4
  15. 2023Active Representation Learning for General Task Space with Applications in Roboticscited by 2
  16. 2024Generative Deduplication For Socia Media Data Selectioncited by 2
  17. 2025Mixtraining: A Better Trade-Off Between Compute and Performancecited by 2
  18. 2022Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Taskscited by 0
  19. 2023Generate to Understand for Representationcited by 0
  20. 2023Source Prompt: Coordinated Pre-training of Language Models on Diverse Corpora from Multiple Sourcescited by 0
  21. 2025Adaptive Sparse Softmax: An Effective and Efficient Softmax Variantcited by 0
  22. 2025Beyond Centralization: Provable Communication Efficient Decentralized Multi-Task Learningcited by 0