mlsanityData to 14 Oct 2026

WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries

arXiv 2407.17468 ↗First posted 24 Jul 2024cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

While hallucinations of large language models (LLMs) prevail as a major challenge, existing evaluation benchmarks on factuality do not cover the diverse domains of knowledge that the real-world users of LLMs seek information about. To bridge this gap, we introduce WildHallucinations, a benchmark that evaluates factuality. It does so by prompting LLMs to generate information about entities mined from user-chatbot conversations in the wild. These generations are then automatically fact-checked against a systematically curated knowledge source collected from web search. Notably, half of these real-world entities do not have associated Wikipedia pages. We evaluate 118,785 generations from 15 LLMs on 7,919 entities. We find that LLMs consistently hallucinate more on entities without Wikipedia pages and exhibit varying hallucination rates across different domains. Finally, given the same base models, adding a retrieval component only slightly reduces hallucinations but does not eliminate hallucinations.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

21 of 28 references.

References · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Mistral 7Bcited by 3,164
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  7. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  8. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  9. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  11. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  12. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
9 more
  1. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  2. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  3. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  4. 2023The Shifted and The Overlooked: A Task-oriented Investigation of User-GPT Interactionscited by 45
  5. 2024HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wildcited by 27
  6. 2024Worse than Random? An Embarrassingly Simple Probing Evaluation of Large Multimodal Models in Medical VQAcited by 19
  7. 2024Large Language Models and User Trust: Consequence of Self-Referential Learning Loop and the Deskilling of Healthcare Professionalscited by 13
  8. 2020Disinformation in the Online Information Ecosystem: Detection, Mitigation and Challengescited by 3
  9. 2021The Rise and Fall of Fake News sites: A Traffic Analysiscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 13.

26 papers.

Sort:

By area:

  1. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  2. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 50
  3. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  4. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  5. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  6. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 13
  7. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  8. 2026HalluHard: A Hard Multi-Turn Hallucination Benchmarkcited by 9
  9. 2025Language Models Can Predict Their Own Behaviorcited by 8
  10. 2025Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrievalcited by 7
  11. 2026To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Callingcited by 7
  12. 2026ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactionscited by 7
14 more
  1. 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
  2. 2026ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Modelscited by 6
  3. 2025Balancing Truthfulness and Informativeness with Uncertainty-Aware Instruction Fine-Tuningcited by 3
  4. 2025Rare Disease Differential Diagnosis with Large Language Models at Scale: From Abdominal Actinomycosis to Wilson's Diseasecited by 2
  5. 2025Context-Aware Human Behavior Prediction Using Multimodal Large Language Models: Challenges and Insightscited by 2
  6. 2026Leveraging LLM Parametric Knowledge for Fact Checking without Retrievalcited by 1
  7. 2026Understanding Why Language Models Hallucinate: Testing Reasoning Against Priorscited by 1
  8. 2025Audited Reasoning Refinement: Fine-Tuning Language Models via LLM-Guided Step-Wise Evaluation and Correctioncited by 0
  9. 2026The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluationcited by 0
  10. 2026Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMscited by 0
  11. 2026Per-Entity Bias Mapping for AI Visibility: Why Brand Mentions Require Entity-Specific Calibrationcited by 0
  12. 2026Reliability Scales Inversely: Hallucinations Snowball Faster in Bigger Language Modelscited by 0
  13. 2026Stick to What You Know: A Study of Knowledge-Aligned Supervised Fine-Tuningcited by 0
  14. 2026FrankenReport: Early Exiting in Long-Form Generation Using Expected Value of Computationcited by 0