mlsanityData to 14 Oct 2026

ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Models

arXiv 2303.16421 ↗First posted 29 Mar 2023cs.CLcited by 63

Not on the map. Why?

Cited by 23 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have made significant progress in NLP. However, their ability to memorize, represent, and leverage commonsense knowledge has been a well-known pain point. In this paper, we specifically focus on ChatGPT, a widely used and easily accessible LLM, and ask the following questions: (1) Can ChatGPT effectively answer commonsense questions? (2) Is ChatGPT aware of the underlying commonsense knowledge for answering a specific question? (3) Is ChatGPT knowledgeable in commonsense? (4) Can ChatGPT effectively leverage commonsense for answering questions? We conduct a series of experiments on 11 datasets to evaluate ChatGPT's commonsense abilities, including answering commonsense questions, identifying necessary knowledge, generating knowledge descriptions, and using knowledge descriptions to answer questions again. Experimental results show that: (1) ChatGPT can achieve good QA accuracies in commonsense tasks, while still struggling with certain domains of datasets. (2) ChatGPT is knowledgeable, and can accurately generate most of the commonsense knowledge using knowledge prompts. (3) Despite its knowledge, ChatGPT is an inexperienced commonsense problem solver, which cannot precisely identify the needed commonsense for answering a specific question. These findings raise the need to explore improved mechanisms for effectively incorporating commonsense into LLMs like ChatGPT, such as better instruction following and commonsense guidance.

Key facts. Cited by 63 papers in the map (data through 14 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 29 Mar 2023.

Builds on

41 of 62 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  12. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
29 more
  1. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  2. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  3. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 214
  4. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  5. 2020Unsupervised Commonsense Question Answering with Self-Talkcited by 161
  6. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 123
  7. 2020Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Modelscited by 110
  8. 2020KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoningcited by 106
  9. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  10. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  11. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  12. 2018Commonsense for Generative Multi-Hop Question Answering Taskscited by 48
  13. 2020ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoningcited by 46
  14. 2020Connecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answeringcited by 44
  15. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  16. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  17. 2020Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoningcited by 33
  18. 2021"I'm Not Mad": Commonsense Implications of Negation and Contradictioncited by 26
  19. 2020On Commonsense Cues in BERT for Solving Commonsense Taskscited by 13
  20. 2022Do language models have coherent mental models of everyday things?cited by 13
  21. 2021Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Modelscited by 11
  22. 2021A Semantic-based Method for Unsupervised Commonsense Question Answeringcited by 10
  23. 2021Do Language Models Perform Generalizable Commonsense Inference?cited by 10
  24. 2021Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledgecited by 8
  25. 2020Commonsense Evidence Generation and Injection in Reading Comprehensioncited by 7
  26. 2021Toward a New Science of Common Sensecited by 6
  27. 2021Alleviating the Knowledge-Language Inconsistency: A Study for Deep Commonsense Knowledgecited by 4
  28. 2021Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Modelscited by 4
  29. 2022TSGP: Two-Stage Generative Prompting for Unsupervised Commonsense Question Answeringcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 36; 2024: 20; 2025: 5; 2026: 2.

63 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Modelscited by 2,576
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  4. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 309
  5. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  6. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 190
  7. 2023ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generationcited by 168
  8. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  9. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 160
  10. 2023Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulnesscited by 130
  11. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
  12. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
38 more
  1. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  2. 2023Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and Beyondcited by 65
  3. 2023GeoLLM: Extracting Geospatial Knowledge from Large Language Modelscited by 64
  4. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  5. 2023Watermarking Conditional Text Generation for AI Detection: Unveiling Challenges and a Semantic-Aware Watermark Remedycited by 50
  6. 2023Tree-Planner: Efficient Close-loop Task Planning with Large Language Modelscited by 48
  7. 2023An Empirical Study on Information Extraction using Large Language Modelscited by 46
  8. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
  9. 2023Fighting Fire with Fire: Can ChatGPT Detect AI-generated Text?cited by 39
  10. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 31
  11. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  12. 2024Strong and weak alignment of large language models with human valuescited by 27
  13. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  14. 2024Democratizing Fine-grained Visual Recognition with Large Language Modelscited by 25
  15. 2023How susceptible are LLMs to Logical Fallacies?cited by 23
  16. 2024When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Modelscited by 22
  17. 2024Large Language Models are Zero-Shot Next Location Predictorscited by 22
  18. 2023Learning to Reason over Scene Graphs: A Case Study of Finetuning GPT-2 into a Robot Language Model for Grounded Task Planningcited by 16
  19. 2023Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Modelscited by 16
  20. 2024Benchmarking Chinese Commonsense Reasoning of LLMs: From Chinese-Specifics to Reasoning-Memorization Correlationscited by 14
  21. 2023Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoningcited by 13
  22. 2024KnowTuning: Knowledge-aware Fine-tuning for Large Language Modelscited by 11
  23. 2024Hybrid Reasoning Based on Large Language Models for Autonomous Car Drivingcited by 11
  24. 2024Right for Right Reasons: Large Language Models for Verifiable Commonsense Knowledge Graph Question Answeringcited by 11
  25. 2024MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Datasetcited by 11
  26. 2023An Overview Of Temporal Commonsense Reasoning and Acquisitioncited by 10
  27. 2024Can LLMs perform structured graph reasoning?cited by 9
  28. 2024NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updatescited by 8
  29. 2024Can generative AI and ChatGPT outperform humans on cognitive-demanding problem-solving tasks in science?cited by 7
  30. 2023EvEval: A Comprehensive Evaluation of Event Semantics for Large Language Modelscited by 6
  31. 2023A New Dialogue Response Generation Agent for Large Language Models by Asking Questions to Detect User's Intentionscited by 6
  32. 2024ConstraintChecker: A Plugin for Large Language Models to Reason on Commonsense Knowledge Basescited by 6
  33. 2023Enhancing In-Context Learning with Answer Feedback for Multi-Span Question Answeringcited by 5
  34. 2023The Two Word Test: A Semantic Benchmark for Large Language Modelscited by 5
  35. 2024A Comprehensive Evaluation on Event Reasoning of Large Language Modelscited by 4
  36. 2025Solving Situation Puzzles with Large Language Model and External Reformulationcited by 4
  37. 2023From Military to Healthcare: Adopting and Expanding Ethical Principles for Generative Artificial Intelligencecited by 3
  38. 2023A Pipeline For Discourse Circuits From CCGcited by 3

50 most cited of 63 All 63 (JSON)