# A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

arXiv: https://arxiv.org/abs/2310.12321 · HTML: https://mlsanity.com/paper/2310.12321/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 99 of its 146 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Builds on (most cited first, 30 of 99)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)

## Built on by (most cited first, 30 of 81)

- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [Evaluating Human-AI Collaboration: A Review and Methodological Framework](https://mlsanity.com/paper/2407.19098/) (2024)
- [Surveying the MLLM Landscape: A Meta-Review of Current Surveys](https://mlsanity.com/paper/2409.18991/) (2024)
- [CoachLM: Automatic Instruction Revisions Improve the Data Quality in LLM Instruction Tuning](https://mlsanity.com/paper/2311.13246/) (2023)
- [Defining Foundation Models for Computational Science: A Call for Clarity and Rigor](https://mlsanity.com/paper/2505.22904/) (2025)
- [LogLM: From Task-based to Instruction-based Automated Log Analysis](https://mlsanity.com/paper/2410.09352/) (2024)
- [DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference](https://mlsanity.com/paper/2501.10375/) (2024)
- [Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning](https://mlsanity.com/paper/2405.16933/) (2024)
- [UPAM: Unified Prompt Attack in Text-to-Image Generation Models Against Both Textual Filters and Visual Checkers](https://mlsanity.com/paper/2405.11336/) (2024)
- [Fairness Definitions in Language Models Explained](https://mlsanity.com/paper/2407.18454/) (2024)
- [GPT for Games: A Scoping Review (2020-2023)](https://mlsanity.com/paper/2404.17794/) (2024)
- [How Can I Improve? Using GPT to Highlight the Desired and Undesired Parts of Open-ended Responses](https://mlsanity.com/paper/2405.00291/) (2024)
- [GPT for Games: An Updated Scoping Review (2020-2024)](https://mlsanity.com/paper/2411.00308/) (2024)
- [Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format](https://mlsanity.com/paper/2411.15982/) (2024)
- [Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategies](https://mlsanity.com/paper/2501.08441/) (2025)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [Comprehensive Evaluation and Insights into the Use of Large Language Models in the Automation of Behavior-Driven Development Acceptance Test Formulation](https://mlsanity.com/paper/2403.14965/) (2024)
- [Leveraging Generative AI for Urban Digital Twins: A Scoping Review on the Autonomous Generation of Urban Data, Scenarios, Designs, and 3D City Models for Smart City Advancement](https://mlsanity.com/paper/2405.19464/) (2024)
- [On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards](https://mlsanity.com/paper/2407.04065/) (2024)
- [Adapt-Pruner: Adaptive Structural Pruning for Efficient Small Language Model Training](https://mlsanity.com/paper/2502.03460/) (2025)
- [Fine-Tuning or Fine-Failing? Debunking Performance Myths in Large Language Models](https://mlsanity.com/paper/2406.11201/) (2024)
- [Towards Advancing Code Generation with Large Language Models: A Research Roadmap](https://mlsanity.com/paper/2501.11354/) (2025)
- [Unified Prompt Attack Against Text-to-Image Generation Models](https://mlsanity.com/paper/2502.16423/) (2025)
- [CHIME: A Case for Efficient Long-Context Attention-FC Disaggregated Inference with DIMM-PIM](https://mlsanity.com/paper/2504.17584/) (2025)
- [Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition](https://mlsanity.com/paper/2508.08274/) (2025)
- [Who Owns the Output? Bridging Law and Technology in LLMs Attribution](https://mlsanity.com/paper/2504.01032/) (2025)
- [Foundation Models for Environmental Science: A Survey of Emerging Frontiers](https://mlsanity.com/paper/2504.04280/) (2025)
- [LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models](https://mlsanity.com/paper/2505.15293/) (2025)
- [Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations](https://mlsanity.com/paper/2404.05415/) (2024)
- [scReader: Prompting Large Language Models to Interpret scRNA-seq Data](https://mlsanity.com/paper/2412.18156/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are a special class of pretrained language models obtained by scaling model size, pretraining corpus and computation. LLMs, because of their large size and pretraining on large volumes of text data, exhibit special abilities which allow them to achieve remarkable performances without any task-specific training in many of the natural language processing tasks. The era of LLMs started with OpenAI GPT-3 model, and the popularity of LLMs is increasing exponentially after the introduction of models like ChatGPT and GPT4. We refer to GPT-3 and its successor OpenAI models, including ChatGPT and GPT4, as GPT-3 family large language models (GLLMs). With the ever-rising popularity of GLLMs, especially in the research community, there is a strong need for a comprehensive survey which summarizes the recent research progress in multiple dimensions and can guide the research community with insightful future research directions. We start the survey paper with foundation concepts like transformers, transfer learning, self-supervised learning, pretrained language models and large language models. We then present a brief overview of GLLMs and discuss the performances of GLLMs in various downstream tasks, specific domains and multiple languages. We also discuss the data labelling and data augmentation abilities of GLLMs, the robustness of GLLMs, the effectiveness of GLLMs as evaluators, and finally, conclude with multiple insightful future research directions. To summarize, this comprehensive survey paper will serve as a good resource for both academic and industry people to stay updated with the latest research related to GPT-3 family large language models.