mlsanityData to 7 Oct 2026

Pre-Trained Models: Past, Present and Future

arXiv 2106.07139 ↗First posted 14 Jun 2021cs.AI cs.CLcited by 319

Not on the map. Why?

Abstract (arXiv)

Large-scale pre-trained models (PTMs) such as BERT and GPT have recently achieved great success and become a milestone in the field of artificial intelligence (AI). Owing to sophisticated pre-training objectives and huge model parameters, large-scale PTMs can effectively capture knowledge from massive labeled and unlabeled data. By storing knowledge into huge parameters and fine-tuning on specific tasks, the rich knowledge implicitly encoded in huge parameters can benefit a variety of downstream tasks, which has been extensively demonstrated via experimental verification and empirical analysis. It is now the consensus of the AI community to adopt PTMs as backbone for downstream tasks rather than learning models from scratch. In this paper, we take a deep look into the history of pre-training, especially its special relation with transfer learning and self-supervised learning, to reveal the crucial position of PTMs in the AI development spectrum. Further, we comprehensively review the latest breakthroughs of PTMs. These breakthroughs are driven by the surge of computational power and the increasing availability of data, towards four important directions: designing effective architectures, utilizing rich contexts, improving computational efficiency, and conducting interpretation and theoretical analysis. Finally, we discuss a series of open problems and research directions of PTMs, and hope our view can inspire and advance the future study of PTMs.

Key facts. Cited by 319 papers in the map (data through 7 Oct 2026). Builds on 230 of its 321 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

230 of 321 references.

References · 230

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  12. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
38 more
  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2017Graph Attention Networkscited by 8,854
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2016Layer Normalizationcited by 5,579
  13. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  14. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  15. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  16. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  17. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  18. 2014Going Deeper with Convolutionscited by 4,231
  19. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  20. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  21. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  22. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  23. 2020Longformer: The Long-Document Transformercited by 3,362
  24. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  25. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  26. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  27. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  28. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  29. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  30. 2015VQA: Visual Question Answeringcited by 2,489
  31. 2018Deep contextualized word representationscited by 2,285
  32. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  33. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  34. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  35. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  36. 2019Language Models as Knowledge Bases?cited by 1,922
  37. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  38. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830

50 most cited of 230 All 230 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 70; 2023: 80; 2024: 86; 2025: 42; 2026: 19.

319 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2024Large Language Models: A Surveycited by 506
  5. 2022Transformers in Time Series: A Surveycited by 489
  6. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  7. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2023Tool Learning with Foundation Modelscited by 302
  10. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  11. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  12. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
38 more
  1. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  2. 2021Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classificationcited by 183
  3. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  4. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  5. 2023Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approachcited by 168
  6. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  7. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  8. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  9. 2021OpenPrompt: An Open-source Framework for Prompt-learningcited by 148
  10. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  11. 2023Auditing large language models: a three-layered approachcited by 135
  12. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  13. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  14. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  15. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  16. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  17. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  18. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  19. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  20. 2024Position: Graph Foundation Models are Already Herecited by 84
  21. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  22. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  23. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  24. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  25. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  26. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 74
  27. 2022OntoProtein: Protein Pretraining With Gene Ontology Embeddingcited by 70
  28. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  29. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  30. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  31. 2023PILOT: A Pre-Trained Model-Based Continual Learning Toolboxcited by 64
  32. 2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
  33. 2024Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysiscited by 60
  34. 2023AI-Generated Content (AIGC): A Surveycited by 59
  35. 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 58
  36. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  37. 2022Prototypical Verbalizer for Prompt-based Few-shot Tuningcited by 53
  38. 2021Complex Knowledge Base Question Answering: A Surveycited by 52

50 most cited of 319 All 319 (JSON)