mlsanityData to 7 Oct 2026

A Categorical Archive of ChatGPT Failures

arXiv 2302.03494 ↗First posted 6 Feb 2023cs.CL cs.AI cs.LGcited by 167

Not on the map. Why?

Abstract (arXiv)

Large language models have been demonstrated to be valuable in different fields. ChatGPT, developed by OpenAI, has been trained using massive amounts of data and simulates human conversation by comprehending context and generating appropriate responses. It has garnered significant attention due to its ability to effectively answer a broad range of human inquiries, with fluent and comprehensive answers surpassing prior public chatbots in both security and usefulness. However, a comprehensive analysis of ChatGPT's failures is lacking, which is the focus of this study. Eleven categories of failures, including reasoning, factual errors, math, coding, and bias, are presented and discussed. The risks, limitations, and societal implications of ChatGPT are also highlighted. The goal of this study is to assist researchers and developers in enhancing future language models and chatbots.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Builds on 34 of its 73 references that are in the map. First posted on arXiv 6 Feb 2023.

Builds on

34 of 73 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  11. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
22 more
  1. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  3. 2023Extracting Training Data from Diffusion Modelscited by 723
  4. 2023A Watermark for Large Language Modelscited by 667
  5. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  6. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  7. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  8. 2022A Systematic Evaluation of Large Language Models of Codecited by 360
  9. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  10. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  11. 2018Pros and Cons of GAN Evaluation Measurescited by 224
  12. 2022Can large language models reason about medical questions?cited by 194
  13. 2023The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientationcited by 183
  14. 2021Challenges in Detoxifying Language Modelscited by 169
  15. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  16. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 121
  17. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 108
  18. 2021Sustainable Modular Debiasing of Language Modelscited by 98
  19. 2022ChatGPT: The End of Online Exam Integrity?cited by 88
  20. 2023Putting ChatGPT's Medical Advice to the (Turing) Testcited by 23
  21. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  22. 2023The moral authority of ChatGPTcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 105; 2024: 38; 2025: 17; 2026: 7.

167 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023Tool Learning with Foundation Modelscited by 302
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  6. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  7. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  8. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  9. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 152
  10. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  11. 2023Auditing large language models: a three-layered approachcited by 135
  12. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
38 more
  1. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  2. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  3. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  4. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  5. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  6. 2023Testing of Detection Tools for AI-Generated Textcited by 86
  7. 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 86
  8. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  9. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  10. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  11. 2023ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Designcited by 66
  12. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  13. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  14. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  15. 2023Unveiling Security, Privacy, and Ethical Concerns of ChatGPTcited by 53
  16. 2023Zero-shot Temporal Relation Extraction with ChatGPTcited by 52
  17. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 50
  18. 2023VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototypingcited by 46
  19. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  20. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  21. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  22. 2024Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Modelscited by 44
  23. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  24. 2023Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questionscited by 42
  25. 2023Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPTcited by 41
  26. 2023Debate Helps Supervise Unreliable Expertscited by 39
  27. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35
  28. 2023Trapping LLM Hallucinations Using Tagged Context Promptscited by 34
  29. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  30. 2023Towards Human-Bot Collaborative Software Architecting with ChatGPTcited by 31
  31. 2023Qualitative Failures of Image Generation Models and Their Application in Detecting Deepfakescited by 31
  32. 2023Interactive Natural Language Processingcited by 31
  33. 2024OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMscited by 28
  34. 2023ChatGPT for Programming Numerical Methodscited by 26
  35. 2023DataTales: Investigating the use of Large Language Models for Authoring Data-Driven Articlescited by 24
  36. 2024Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seekingcited by 24
  37. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  38. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24

50 most cited of 167 All 167 (JSON)