mlsanityData to 7 Oct 2026

Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

arXiv 2004.10964 ↗First posted 23 Apr 2020cs.CL cs.LGcited by 1,548

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models pretrained on text from a wide variety of sources form the foundation of today's NLP. In light of the success of these broad-coverage models, we investigate whether it is still helpful to tailor a pretrained model to the domain of a target task. We present a study across four domains (biomedical and computer science publications, news, and reviews) and eight classification tasks, showing that a second phase of pretraining in-domain (domain-adaptive pretraining) leads to performance gains, under both high- and low-resource settings. Moreover, adapting to the task's unlabeled data (task-adaptive pretraining) improves performance even after domain-adaptive pretraining. Finally, we show that adapting to a task corpus augmented using simple data selection strategies is an effective alternative, especially when resources for domain-adaptive pretraining might be unavailable. Overall, we consistently find that multi-phase adaptive pretraining offers large gains in task performance.

Key facts. Cited by 1,548 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 260; 2022: 280; 2023: 308; 2024: 282; 2025: 217; 2026: 201.

1,548 papers.

Sort:

By area:

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  3. 2023BloombergGPT: A Large Language Model for Financecited by 923
  4. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  5. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  6. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  9. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  10. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  11. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  12. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
38 more
  1. 2022One Embedder, Any Task: Instruction-Finetuned Text Embeddingscited by 329
  2. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  5. 2022Generating Sequences by Learning to Self-Correctcited by 271
  6. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  7. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  9. 2022Data Contamination: From Memorization to Exploitationcited by 234
  10. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  13. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  14. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  15. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  16. 2023Continual Pre-training of Language Modelscited by 194
  17. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  18. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  19. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  20. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  21. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  22. 2024Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performancecited by 179
  23. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  24. 2022Can Foundation Models Wrangle Your Data?cited by 173
  25. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  26. 2021Challenges in Detoxifying Language Modelscited by 169
  27. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  28. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168
  29. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  30. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  31. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  32. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 159
  33. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  34. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 153
  35. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  36. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  37. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  38. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146

50 most cited of 1,548 All 1,548 (JSON)