mlsanityData to 7 Oct 2026

Why Does ChatGPT Fall Short in Providing Truthful Answers?

arXiv 2304.10513 ↗First posted 20 Apr 2023cs.CL cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

Recent advancements in large language models, such as ChatGPT, have demonstrated significant potential to impact various aspects of human life. However, ChatGPT still faces challenges in providing reliable and accurate answers to user questions. To better understand the model's particular weaknesses in providing truthful answers, we embark an in-depth exploration of open-domain question answering. Specifically, we undertake a detailed examination of ChatGPT's failures, categorized into: comprehension, factuality, specificity, and inference. We further pinpoint factuality as the most contributing failure and identify two critical abilities associated with factuality: knowledge memorization and knowledge recall. Through experiments focusing on factuality, we propose several potential enhancement strategies. Our findings suggest that augmenting the model with granular external knowledge and cues for knowledge recall can enhance the model's factuality in answering questions.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

28 of 38 references.

References · 28

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  11. 2022Decomposed Prompting: A Modular Approach for Solving Complex Taskscited by 554
  12. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
16 more
  1. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  2. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  3. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  4. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  5. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  6. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  7. 2023A Categorical Archive of ChatGPT Failurescited by 167
  8. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  9. 2023Emergent autonomous scientific research capabilities of large language modelscited by 127
  10. 2023An Analysis of the Automatic Bug Fixing Performance of ChatGPTcited by 108
  11. 2022Evaluating the Text-to-SQL Capabilities of Large Language Modelscited by 101
  12. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  13. 2022Multi-hop Question Answeringcited by 64
  14. 2022Read before Generate! Faithful Long Form Question Answering with Machine Readingcited by 47
  15. 2021On the Multilingual Capabilities of Very Large-Scale English Language Modelscited by 22
  16. 2022Can Language Models Be Specific? How?cited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 18; 2025: 10; 2026: 5.

54 papers.

Sort:

By area:

  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  5. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  6. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  7. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  8. 2023Alignment for Honestycited by 72
  9. 2023Large Language Models can Learn Rulescited by 66
  10. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  11. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  12. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
38 more
  1. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 36
  2. 2023Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinationscited by 34
  3. 2023Evaluating Hallucinations in Chinese Large Language Modelscited by 33
  4. 2024A Survey of Useful LLM Evaluationcited by 28
  5. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 26
  6. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  7. 2024Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agentscited by 22
  8. 2023Citation: A Key to Building Responsible and Accountable Large Language Modelscited by 20
  9. 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Frameworkcited by 19
  10. 2024Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparatorscited by 19
  11. 2023The Hitchhiker's Guide to Program Analysis: A Journey with Large Language Modelscited by 18
  12. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  13. 2024Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrievalcited by 16
  14. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  15. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  16. 2023Quantifying and Attributing the Hallucination of Large Language Models via Association Analysiscited by 14
  17. 2024Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Koreancited by 14
  18. 2024Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self-Consistencycited by 14
  19. 2024The Impact of Prompt Programming on Function-Level Code Generationcited by 13
  20. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Modelscited by 12
  21. 2023Examining the Potential and Pitfalls of ChatGPT in Science and Engineering Problem-Solvingcited by 11
  22. 2024REAL Sampling: Boosting Factuality and Diversity of Open-Ended Generation via Asymptotic Entropycited by 11
  23. 2024Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Modelscited by 11
  24. 2023AutoHall: Automated Factuality Hallucination Dataset Generation for Large Language Modelscited by 10
  25. 2023Axiomatic Preference Modeling for Longform Question Answeringcited by 6
  26. 2025How Does Response Length Affect Long-Form Factualitycited by 6
  27. 2024Luna: An Evaluation Foundation Model to Catch Language Model Hallucinations with High Accuracy and Low Costcited by 5
  28. 2024How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questionscited by 5
  29. 2024GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?cited by 3
  30. 2025HIDE and Seek: Detecting Hallucinations in Language Models via Decoupled Representationscited by 2
  31. 2023Leveraging Explainable AI to Analyze Researchers' Aspect-Based Sentiment about ChatGPTcited by 1
  32. 2024AI Assistants for Spaceflight Procedures: Combining Generative Pre-Trained Transformer and Retrieval-Augmented Generation on Knowledge Graphs With Augmented Reality Cuescited by 1
  33. 2025Exploring and Mitigating Fawning Hallucinations in Large Language Modelscited by 1
  34. 2025HalluClean: A Unified Framework to Combat Hallucinations in LLMscited by 1
  35. 2025Teaching People LLM's Errors and Getting it Rightcited by 1
  36. 2026Understanding Why Language Models Hallucinate: Testing Reasoning Against Priorscited by 1
  37. 2023Critical Role of Artificially Intelligent Conversational Chatbotcited by 0
  38. 2025Flippi: End To End GenAI Assistant for E-Commercecited by 0

50 most cited of 54 All 54 (JSON)