mlsanityData to 7 Oct 2026

Pre-trained Models for Natural Language Processing: A Survey

arXiv 2003.08271 ↗First posted 18 Mar 2020cs.CL cs.LGcited by 504

Not on the map. Why?

Abstract (arXiv)

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy with four perspectives. Next, we describe how to adapt the knowledge of PTMs to the downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

Key facts. Cited by 504 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 82; 2022: 128; 2023: 139; 2024: 88; 2025: 44; 2026: 23.

504 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  3. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  4. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  6. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  7. 2024Large Language Models: A Surveycited by 506
  8. 2022Transformers in Time Series: A Surveycited by 489
  9. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  10. 2021A Survey of Transformerscited by 399
  11. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  12. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
38 more
  1. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  2. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  3. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  4. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  7. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  8. 2023AdaMerging: Adaptive Model Merging for Multi-Task Learningcited by 248
  9. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  10. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  11. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  12. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  13. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  14. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  15. 2023OpenAGI: When LLM Meets Domain Expertscited by 196
  16. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  17. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  18. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  19. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  20. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  21. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  22. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  23. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  24. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  25. 2021SCARF: Self-Supervised Contrastive Learning using Random Feature Corruptioncited by 151
  26. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  27. 2024How do Large Language Models Handle Multilingualism?cited by 148
  28. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  29. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  30. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  31. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  32. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  33. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  34. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  35. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  36. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  37. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  38. 2022Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecastingcited by 117

50 most cited of 504 All 504 (JSON)