mlsanityData to 14 Oct 2026

Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions

arXiv 2305.10435 ↗First posted 11 May 2023cs.CL cs.AIcited by 55

Not on the map. Why?

Abstract (arXiv)

The Generative Pre-trained Transformer (GPT) represents a notable breakthrough in the domain of natural language processing, which is propelling us toward the development of machines that can understand and communicate using language in a manner that closely resembles that of humans. GPT is based on the transformer architecture, a deep neural network designed for natural language processing tasks. Due to their impressive performance on natural language processing tasks and ability to effectively converse, GPT have gained significant popularity among researchers and industrial communities, making them one of the most widely used and effective models in natural language processing and related fields, which motivated to conduct this review. This review provides a detailed overview of the GPT, including its architecture, working process, training procedures, enabling technologies, and its impact on various applications. In this review, we also explored the potential challenges and limitations of a GPT. Furthermore, we discuss potential solutions and future directions. Overall, this paper aims to provide a comprehensive understanding of GPT, enabling technologies, their impact on various applications, emerging challenges, and potential solutions.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 73 of its 246 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

73 of 246 references.

References · 73

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  7. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  10. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  11. 2021GPT Understands, Toocited by 716
  12. 2019Adversarial Training for Free!cited by 643
38 more
  1. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  2. 2020Jukebox: A Generative Model for Musiccited by 542
  3. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
  4. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  5. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  6. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  7. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  8. 2021Recent Advances in Adversarial Training for Adversarial Robustnesscited by 279
  9. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  10. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  11. 2023GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Modelscited by 176
  12. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
  13. 2019Challenges in Building Intelligent Open-domain Dialog Systemscited by 147
  14. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  15. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 130
  16. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 119
  17. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  18. 2021WebQA: Multihop and Multimodal QAcited by 109
  19. 2021Jigsaw: Large Language Models meet Program Synthesiscited by 103
  20. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  21. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 90
  22. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  23. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  24. 2022The AI Teacher Test: Measuring the Pedagogical Ability of Blender and GPT-3 in Educational Dialoguescited by 79
  25. 2023AI model GPT-3 (dis)informs us better than humanscited by 78
  26. 2023Unleashing the Power of Edge-Cloud Generative AI in Mobile Networks: A Survey of AIGC Servicescited by 76
  27. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  28. 2022GPT Takes the Bar Examcited by 65
  29. 2016Defensive Distillation is Not Robust to Adversarial Examplescited by 60
  30. 2023Chat2VIS: Generating Data Visualisations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Modelscited by 59
  31. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54
  32. 2023ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishingcited by 52
  33. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 39
  34. 2022On the Explainability of Natural Language Processing Deep Modelscited by 38
  35. 2022Opal: Multimodal Image Generation for News Illustrationcited by 36
  36. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  37. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  38. 2022"I think this is the most disruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Datacited by 32

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 27; 2025: 18; 2026: 6.

55 papers.

Sort:

By area:

  1. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  2. 2023Wireless Multi-Agent Generative AI: From Connected Intelligence to Collective Intelligencecited by 26
  3. 2024Large Language Models for Mobility Analysis in Transportation Systems: A Survey on Forecasting Taskscited by 16
  4. 2024GenAINet: Enabling Wireless Collective Intelligence via Knowledge Transfer and Reasoningcited by 13
  5. 2024ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large Language Modelscited by 11
  6. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problemscited by 10
  7. 2024Large Language Models Playing Mixed Strategy Nash Equilibrium Gamescited by 9
  8. 2024Machine Learning Robustness: A Primercited by 8
  9. 2024AlphaZip: Neural Network-Enhanced Lossless Text Compressioncited by 7
  10. 2024The influence of persona and conversational task on social interactions with a LLM-controlled embodied conversational agentcited by 7
  11. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  12. 2025Evaluation of LLMs for mathematical problem solvingcited by 7
38 more
  1. 2024Generative AI for RF Sensing in IoT systemscited by 6
  2. 2024Towards Better Multi-task Learning: A Framework for Optimizing Dataset Combinations in Large Language Modelscited by 6
  3. 2024ErgoChat: a Visual Query System for the Ergonomic Risk Assessment of Construction Workerscited by 6
  4. 2025IberBench: LLM Evaluation on Iberian Languagescited by 5
  5. 2024Bridging Text and Molecule: A Survey on Multimodal Frameworks for Moleculecited by 4
  6. 2024Unifying Sequences, Structures, and Descriptions for Any-to-Any Protein Generation with the Large Multimodal Model HelixProtXcited by 4
  7. 2024Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curlingcited by 4
  8. 2025Towards the Autonomous Optimization of Urban Logistics: Training Generative AI with Scientific Tools via Agentic Digital Twins and Model Context Protocolcited by 4
  9. 2024Utilizing GPT to Enhance Text Summarization: A Strategy to Minimize Hallucinationscited by 3
  10. 2024Using Large Language Models in Public Transit Systems, San Antonio as a case studycited by 3
  11. 2025Towards an automated workflow in materials science for combining multi-modal simulative and experimental information using data mining and large language modelscited by 3
  12. 2025HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokenscited by 3
  13. 2025Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challengescited by 3
  14. 2024Automatic deductive coding in discourse analysis: an application of large language models in learning analyticscited by 2
  15. 2025Cross-Question Method Reuse in Large Language Models: From Word-Level Prediction to Rational Logical-Layer Reasoningcited by 2
  16. 2026L-PRISMA: An Extension of PRISMA in the Era of Generative Artificial Intelligence (GenAI)cited by 2
  17. 2023Guarding the Guardians: Automated Analysis of Online Child Sexual Abusecited by 1
  18. 2023RetSeg: Retention-based Colorectal Polyps Segmentation Networkcited by 1
  19. 2023DeepFracture: A Generative Approach for Predicting Brittle Fractures with Neural Discrete Representation Learningcited by 1
  20. 2024Spatial Computing: Concept, Applications, Challenges and Future Directionscited by 1
  21. 2024Visual Question Answering in Ophthalmology: A Progressive and Practical Perspectivecited by 1
  22. 2024RetrieveGPT: Merging Prompts and Mathematical Models for Enhanced Code-Mixed Information Retrievalcited by 1
  23. 2025OFCnetLLM: Large Language Model for Network Monitoring and Alertnesscited by 1
  24. 2025Talking Surveys: How Photorealistic Embodied Conversational Agents Shape Response Quality, Engagement, and Satisfactioncited by 1
  25. 2024Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformerscited by 0
  26. 2024Integrating AI in College Education: Positive yet Mixed Experiences with ChatGPTcited by 0
  27. 2024Trading Devil Final: Backdoor attack via Stock market and Bayesian Optimizationcited by 0
  28. 2024Evaluating Large Language Models for automatic analysis of teacher simulationscited by 0
  29. 2024A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patientscited by 0
  30. 2024One Pixel is All I Needcited by 0
  31. 2024Trading Devil RL: Backdoor attack via Stock market, Bayesian Optimization and Reinforcement Learningcited by 0
  32. 2025Balancing Innovation and Integrity: AI Integration in Liberal Arts College Administrationcited by 0
  33. 2025ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoningcited by 0
  34. 2025Distributionally Robust Wireless Semantic Communication with Large AI Modelscited by 0
  35. 2025Assessing the Capabilities and Limitations of FinGPT Model in Financial NLP Applicationscited by 0
  36. 2025CorefInst: Leveraging LLMs for Multilingual Coreference Resolutioncited by 0
  37. 2025Large Language Models for Explainable Threat Intelligencecited by 0
  38. 2025Risk-Aware Financial Forecasting Enhanced by Machine Learning and Intuitionistic Fuzzy Multi-Criteria Decision-Makingcited by 0

50 most cited of 55 All 55 (JSON)