mlsanityData to 7 Oct 2026

A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models

arXiv 2303.10420 ↗First posted 18 Mar 2023cs.CLcited by 244

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.

Key facts. Cited by 244 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 18 Mar 2023.

Builds on

16 of 33 references.

References · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
4 more
  1. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  2. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  3. 2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 120
  4. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 47; 2024: 104; 2025: 79; 2026: 14.

244 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  3. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  4. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  5. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  6. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  7. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 167
  8. 2025MemOS: A Memory OS for AI Systemcited by 129
  9. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  10. 2023Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Casescited by 85
  11. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 84
  12. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
38 more
  1. 2024LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256Kcited by 80
  2. 2024If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agentscited by 77
  3. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  4. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  5. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  6. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 70
  7. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  8. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  9. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  10. 2023Towards Interpretable Mental Health Analysis with Large Language Modelscited by 53
  11. 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 48
  12. 2023Theory of Mind in Large Language Models: Examining Performance of 11 State-of-the-Art models vs. Children Aged 7-10 on Advanced Testscited by 46
  13. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
  14. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  15. 2024Physical Property Understanding from Language-Embedded Feature Fieldscited by 45
  16. 2025ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptionscited by 44
  17. 2023Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5cited by 41
  18. 2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
  19. 2024Large Language Models for Medicine: A Surveycited by 38
  20. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  21. 2023Anatomy of an AI-powered malicious social botnetcited by 35
  22. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  23. 2023Gaining Wisdom from Setbacks: Aligning Large Language Models via Mistake Analysiscited by 34
  24. 2024MATHSENSEI: A Tool-Augmented Large Language Model for Mathematical Reasoningcited by 34
  25. 2023Evaluation of ChatGPT Model for Vulnerability Detectioncited by 33
  26. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  27. 2024LLM4AD: A Platform for Algorithm Design with Large Language Modelcited by 33
  28. 2024Metaphor Understanding Challenge Dataset for LLMscited by 30
  29. 2024Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustmentcited by 29
  30. 2024FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowchartscited by 27
  31. 2023IDEAL: Influence-Driven Selective Annotations Empower In-Context Learners in Large Language Modelscited by 26
  32. 2024Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmarkcited by 26
  33. 2023A Zero-shot and Few-shot Study of Instruction-Finetuned Large Language Models Applied to Clinical and Biomedical Taskscited by 25
  34. 2024DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledgecited by 24
  35. 2024Can Large Language Models perform Relation-based Argument Mining?cited by 23
  36. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  37. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntaxcited by 22
  38. 2024Keypoint-based Progressive Chain-of-Thought Distillation for LLMscited by 22

50 most cited of 244 All 244 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  2. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond
  3. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts
  4. 2025GPT as a Monte Carlo Language Tree: A Probabilistic Perspective
  5. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
  6. 2023Gpt-4: A Review on Advancements and Opportunities in Natural Language Processing
  7. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
  8. 2022Prompting GPT-3 To Be Reliable