mlsanityData to 7 Oct 2026

Do Language Models Know When They're Hallucinating References?

arXiv 2305.18248 ↗First posted 29 May 2023cs.CL cs.AIcited by 115

Not on the map. Why?

Abstract (arXiv)

State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 17 of its 39 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

17 of 39 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2019How Can We Know What Language Models Know?cited by 953
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  10. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  11. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  12. 2021A Survey on Automated Fact-Checkingcited by 376
5 more
  1. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  2. 2023Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runscited by 52
  3. 2018What are the biases in my word embedding?cited by 30
  4. 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
  5. 2020Automatic generation of reviews of scientific paperscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 50; 2025: 26; 2026: 18.

115 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2024Measuring short-form factuality in large language modelscited by 345
  4. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  5. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  6. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  7. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  8. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  9. 2023Fine-tuning Language Models for Factualitycited by 219
  10. 2025Why Language Models Hallucinatecited by 205
  11. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  12. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
38 more
  1. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  2. 2023The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediationscited by 130
  3. 2023Calibrated Language Models Must Hallucinatecited by 114
  4. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  5. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  6. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  7. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  8. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  9. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 92
  10. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinementcited by 89
  11. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  12. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 87
  13. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  14. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  15. 2023A Survey of Large Language Models Attributioncited by 71
  16. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  17. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  18. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45
  19. 2024Learning to Trust Your Feelings: Leveraging Self-awareness in LLMs for Hallucination Mitigationcited by 43
  20. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  21. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
  22. 2024Can Language Models Perform Robust Reasoning in Chain-of-thought Prompting with Noisy Rationales?cited by 40
  23. 2023Ever: Mitigating Hallucination in Large Language Models through Real-Time Verification and Rectificationcited by 38
  24. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Promptscited by 36
  25. 2023Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Trainingcited by 31
  26. 2024Backtracking Improves Generation Safetycited by 31
  27. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 29
  28. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 29
  29. 2024Rowen: Adaptive Retrieval-Augmented Generation for Hallucination Mitigation in LLMscited by 27
  30. 2024HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wildcited by 27
  31. 2025Hallucination Detection in Large Language Models with Metamorphic Relationscited by 24
  32. 2025Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detectioncited by 18
  33. 2024Semantic Membership Inference Attack against Large Language Modelscited by 17
  34. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 16
  35. 2024Large Language Models for Automated Literature Review: An Evaluation of Reference Generation, Abstract Writing, and Review Compositioncited by 15
  36. 2023Quantifying and Attributing the Hallucination of Large Language Models via Association Analysiscited by 14
  37. 2023Fine-tune Language Models to Approximate Unbiased In-context Learningcited by 12
  38. 2024Reconfidencing LLMs from the Grouping Loss Perspectivecited by 11

50 most cited of 115 All 115 (JSON)