mlsanityData to 7 Oct 2026

Is ChatGPT a General-Purpose Natural Language Processing Task Solver?

arXiv 2302.06476 ↗First posted 8 Feb 2023cs.CL cs.AIcited by 486

Not on the map. Why?

Cited by 46 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Spurred by advancements in scale, large language models (LLMs) have demonstrated the ability to perform a variety of natural language processing (NLP) tasks zero-shot -- i.e., without adaptation on downstream data. Recently, the debut of ChatGPT has drawn a great deal of attention from the natural language processing (NLP) community due to the fact that it can generate high-quality responses to human input and self-correct previous mistakes based on subsequent conversations. However, it is not yet known whether ChatGPT can serve as a generalist model that can perform many NLP tasks zero-shot. In this work, we empirically analyze the zero-shot learning ability of ChatGPT by evaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate both the effectiveness and limitations of the current version of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoning capabilities (e.g., arithmetic reasoning) while it still faces challenges when solving specific tasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.

Key facts. Cited by 486 papers in the map (data through 7 Oct 2026). Builds on 61 of its 99 references that are in the map. First posted on arXiv 8 Feb 2023.

Builds on

61 of 99 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
38 more
  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  13. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  14. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  15. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  16. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  17. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  18. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  19. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  20. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  21. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  22. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  23. 2020Knowledge Graphscited by 619
  24. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  25. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  26. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  27. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  28. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  29. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  30. 2016Solving General Arithmetic Word Problemscited by 353
  31. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  32. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  33. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  34. 2022Large Language Models Are Reasoning Teacherscited by 318
  35. 2022UL2: Unifying Language Learning Paradigmscited by 292
  36. 2022Teaching Small Language Models to Reasoncited by 279
  37. 2022Is GPT-3 a Good Data Annotator?cited by 208
  38. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 238; 2024: 159; 2025: 60; 2026: 29.

486 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  7. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  8. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  9. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  11. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
  12. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
38 more
  1. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  2. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  3. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  5. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  6. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  7. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  8. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  9. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  10. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  11. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  12. 2023Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMscited by 191
  13. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  14. 2024Corrective Retrieval Augmented Generationcited by 187
  15. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  16. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 180
  17. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  18. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  19. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  20. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  21. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  22. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  23. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  24. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  25. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  26. 2023Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentationcited by 146
  27. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  28. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  29. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  30. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  31. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  32. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  33. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 128
  34. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  35. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  36. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  37. 2023Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!cited by 119
  38. 2023Large Language Models as Analogical Reasonerscited by 107

50 most cited of 486 All 486 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023GPT-3.5, GPT-4, or BARD? Evaluating LLMs Reasoning Ability in Zero-Shot Setting and Performance Boosting Through Prompts
  2. 2025Comparative Evaluation of ChatGPT and DeepSeek Across Key NLP Tasks: Strengths, Weaknesses, and Domain-Specific Performance
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
  4. 2024Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment
  5. 2023Assessing the Impact of Prompting Methods on ChatGPT's Mathematical Capabilities
  6. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
  7. 2024ChatGPT Alternative Solutions: Large Language Models Survey
  8. 2023Large Language Models Are Zero-Shot Text Classifiers