mlsanityData to 7 Oct 2026

A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT

arXiv 2302.09419 ↗First posted 18 Feb 2023cs.AI cs.CL cs.LGcited by 364

Not on the map. Why?

Abstract (arXiv)

Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A PFM (e.g., BERT, ChatGPT, and GPT-4) is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. BERT learns bidirectional encoder representations from Transformers, which are trained on large datasets as contextual language models. Similarly, the generative pretrained transformer (GPT) method employs Transformers as the feature extractor and is trained using an autoregressive paradigm on large datasets. Recently, ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few shot prompting. The remarkable achievements of PFM have brought significant breakthroughs to various fields of AI. Numerous studies have proposed different methods, raising the demand for an updated survey. This study provides a comprehensive review of recent research advancements, challenges, and opportunities for PFMs in text, image, graph, as well as other data modalities. The review covers the basic components and existing pretraining methods used in natural language processing, computer vision, and graph learning. Additionally, it explores advanced PFMs used for different data modalities and unified PFMs that consider data quality and quantity. The review also discusses research related to the fundamentals of PFMs, such as model efficiency and compression, security, and privacy. Finally, the study provides key implications, future research directions, challenges, and open problems in the field of PFMs. Overall, this survey aims to shed light on the research of the PFMs on scalability, security, logical reasoning ability, cross-domain learning ability, and the user-friendly interactive ability for artificial general intelligence.

Key facts. Cited by 364 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Feb 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 132; 2024: 114; 2025: 85; 2026: 33.

364 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  5. 2024Large Language Models: A Surveycited by 506
  6. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  7. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  8. 2023Segment Anything Model for Medical Image Analysis: an Experimental Studycited by 291
  9. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 267
  10. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  11. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  12. 2023SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagerycited by 202
38 more
  1. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  2. 2023Weakly Supervised 3D Open-vocabulary Segmentationcited by 133
  3. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  4. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  5. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  6. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  7. 2024Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Modelscited by 114
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  10. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  11. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  12. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  13. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  14. 2024GFT: Graph Foundation Model with Transferable Tree Vocabularycited by 82
  15. 2023Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-Incited by 71
  16. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  17. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  18. 2023The Self-Perception and Political Biases of ChatGPTcited by 66
  19. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  20. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  21. 2023Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMscited by 62
  22. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
  23. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  24. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  25. 2024A Comprehensive Survey of Foundation Models in Medicinecited by 56
  26. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  27. 2023Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluationcited by 52
  28. 2023Large Generative AI Models for Telecom: The Next Big Thing?cited by 52
  29. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
  30. 2024GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-Based Social Abusecited by 50
  31. 2023ChatGPT vs Human-authored Text: Insights into Controllable Text Summarization and Sentence Style Transfercited by 49
  32. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  33. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  34. 2023Radiology-GPT: A Large Language Model for Radiologycited by 44
  35. 2023Are Natural Domain Foundation Models Useful for Medical Image Classification?cited by 44
  36. 2023Can Large Language Models Empower Molecular Property Prediction?cited by 43
  37. 2024Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactionscited by 43
  38. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42

50 most cited of 364 All 364 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  2. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
  3. 2023Shall We Pretrain Autoregressive Language Models with Retrieval? A Comprehensive Study
  4. 2026Reliable and Responsible Foundation Models: A Comprehensive Survey
  5. 2025GPT as a Monte Carlo Language Tree: A Probabilistic Perspective
  6. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
  8. 2025Retrieval-augmented Prompt Learning for Pre-trained Foundation Models