mlsanityData to 7 Oct 2026

ChatGPT Hallucinates when Attributing Answers

arXiv 2309.09401 ↗First posted 17 Sep 2023cs.AI cs.DL cs.IRcited by 14

Not on the map. Why?

Abstract (arXiv)

Can ChatGPT provide evidence to support its answers? Does the evidence it suggests actually exist and does it really support its answer? We investigate these questions using a collection of domain-specific knowledge-based questions, specifically prompting ChatGPT to provide both an answer and supporting evidence in the form of references to external sources. We also investigate how different prompts impact answers and evidence. We find that ChatGPT provides correct or partially correct answers in about half of the cases (50.6% of the times), but its suggested references only exist 14% of the times. We further provide insights on the generated references that reveal common traits among the references that ChatGPT generates, and show how even if a reference provided by the model does exist, this reference often does not support the claims ChatGPT attributes to it. Our findings are important because (1) they are the first systematic analysis of the references created by ChatGPT in its answers; (2) they suggest that the model may leverage good quality information in producing correct answers, but is unable to attribute real evidence to support its answers. Prompts, raw result files and manual analysis are made publicly available.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 18 of its 31 references that are in the map. First posted on arXiv 17 Sep 2023.

Builds on

18 of 31 references.

References · 18

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  7. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  10. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  11. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  12. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 85
6 more
  1. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  2. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  3. 2023Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferencescited by 25
  4. 2021Rethinking Search: Making Domain Experts out of Dilettantescited by 18
  5. 2023Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctnesscited by 15
  6. 2022AgAsk: An Agent to Help Answer Farmer's Questions From Scientific Documentscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 7; 2025: 3; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 275
  2. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  3. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 29
  4. 2024AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attributioncited by 23
  5. 2024Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perceptioncited by 17
  6. 2024ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Modelscited by 9
  7. 2025Can LLMs Automate Fact-Checking Article Writing?cited by 7
  8. 2026Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attributioncited by 2
  9. 2023Sequencing Matters: A Generate-Retrieve-Generate Model for Building Conversational Agentscited by 1
  10. 2024Inner-Probe: Discovering Copyright-related Data Generation in LLM Architecturecited by 1
  11. 2025SAFE: Improving LLM Systems using Sentence-Level In-generation Attributioncited by 1
  12. 2025Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Modelscited by 1
2 more
  1. 2026Probing for Knowledge Attribution in Large Language Modelscited by 1
  2. 2026QChunker: Learning Question-Aware Text Chunking for Domain RAG via Multi-Agent Debatecited by 0