mlsanityData to 14 Oct 2026

HalluLens: LLM Hallucination Benchmark

arXiv 2504.17550 ↗First posted 24 Apr 2025cs.CL cs.AIcited by 92

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as "hallucination." These hallucinations undermine user trust and hinder the adoption of generative AI systems. Addressing hallucinations is essential for the advancement of LLMs. This paper introduces a comprehensive hallucination benchmark, incorporating both new extrinsic and existing intrinsic evaluation tasks, built upon clear taxonomy of hallucination. A major challenge in benchmarking hallucinations is the lack of a unified framework due to inconsistent definitions and categorizations. We disentangle LLM hallucination from "factuality," proposing a clear taxonomy that distinguishes between extrinsic and intrinsic hallucinations, to promote consistency and facilitate research. Extrinsic hallucinations, where the generated content is not consistent with the training data, are increasingly important as LLMs evolve. Our benchmark includes dynamic test set generation to mitigate data leakage and ensure robustness against such leakage. We also analyze existing benchmarks, highlighting their limitations and saturation. The work aims to: (1) establish a clear taxonomy of hallucinations, (2) introduce new extrinsic hallucination tasks, with data that can be dynamically regenerated to prevent saturation by leakage, (3) provide a comprehensive analysis of existing benchmarks, distinguishing them from factuality evaluations.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Builds on 60 of its 81 references that are in the map. First posted on arXiv 24 Apr 2025.

Builds on

60 of 81 references.

References · 60

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  12. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
38 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,049
  2. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  5. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  6. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  7. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  8. 2024Measuring short-form factuality in large language modelscited by 373
  9. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  10. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  11. 2016NewsQA: A Machine Comprehension Datasetcited by 331
  12. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  13. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  14. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  15. 2022RealTime QA: What's the Answer Right Now?cited by 244
  16. 2023Do Large Language Models Know What They Don't Know?cited by 243
  17. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 232
  18. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 225
  19. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  20. 2023Mitigating the Alignment Tax of RLHFcited by 183
  21. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
  22. 2022Large Language Models with Controllable Working Memorycited by 169
  23. 2024Long-form factuality in large language modelscited by 155
  24. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 150
  25. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  26. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  27. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  28. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  29. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  30. 2022SciFact-Open: Towards open-domain scientific claim verificationcited by 94
  31. 2024The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Modelscited by 88
  32. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  33. 2020Controlled Hallucinations: Learning to Generate Faithfully from Noisy Datacited by 84
  34. 2024Factuality of Large Language Models: A Surveycited by 66
  35. 2023Knowledge of Knowledge: Exploring Known-Unknowns Uncertainty with Large Language Modelscited by 61
  36. 2022Entity Cloze By Date: What LMs Know About Unseen Entitiescited by 55
  37. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 54
  38. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 36; 2026: 56.

92 papers.

Sort:

By area:

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 145
  2. 2025Embodied AI Agents: Modeling the Worldcited by 61
  3. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 49
  4. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 27
  5. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Themcited by 16
  6. 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systemscited by 14
  7. 2025Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yetcited by 11
  8. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  9. 2026ERNIE 5.0 Technical Reportcited by 11
  10. 2026AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitorcited by 10
  11. 2025Mitigating Manipulation and Enhancing Persuasion: A Reflective Multi-Agent Approach for Legal Argument Generationcited by 8
  12. 2025FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Financecited by 8
38 more
  1. 2025FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factualitycited by 6
  2. 2025PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatrycited by 6
  3. 2026NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systemscited by 6
  4. 2025Beyond Prompt-Induced Lies: Investigating LLM Deception on Benign Promptscited by 5
  5. 2026Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Biascited by 5
  6. 2026Pixels Don't Lie (But Your Detector Might): Bootstrapping MLLM-as-a-Judge for Trustworthy Deepfake Detection and Reasoning Supervisioncited by 5
  7. 2026TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systemscited by 5
  8. 2025Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretationscited by 4
  9. 2025From Noise to Narrative: Tracing the Origins of Hallucinations in Transformerscited by 4
  10. 2025Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factualitycited by 4
  11. 2025Past, Present, and Future of Bug Tracking in the Generative AI Eracited by 4
  12. 2026Hallucinations Live in Variancecited by 4
  13. 2025Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capabilitycited by 3
  14. 2025Structured Prompts Improve Evaluation of Language Modelscited by 3
  15. 2025Latent Debate: A Surrogate Framework for Interpreting LLM Thinkingcited by 3
  16. 2026HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agentscited by 3
  17. 2025MultiHal: Multilingual Dataset for Knowledge-Graph Grounded Evaluation of LLM Hallucinationscited by 2
  18. 2025How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAGcited by 2
  19. 2025AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Researchcited by 2
  20. 2025Confabulations from ACL Publications (CAP): A Dataset for Scientific Hallucination Detectioncited by 2
  21. 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systemscited by 2
  22. 2025Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Explorationcited by 2
  23. 2026Distill and Align Decomposition for Enhanced Claim Verificationcited by 2
  24. 20266G Needs Agents: Toward Agentic AI-Native Networks for Autonomous Intelligencecited by 2
  25. 2026Hallucination as Exploit: Evidence-Carrying Multimodal Agentscited by 2
  26. 2026HalluWorld: A Controlled Benchmark for Hallucination via Reference World Modelscited by 2
  27. 2026PhantomBench: Benchmarking the Non-existential Threat of Language Modelscited by 2
  28. 2026From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narrativescited by 2
  29. 2025Phare: A Safety Probe for Large Language Modelscited by 1
  30. 2025SAFE: Improving LLM Systems using Sentence-Level In-generation Attributioncited by 1
  31. 2025RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrievalcited by 1
  32. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  33. 2025HalluGen: Synthesizing Realistic and Controllable Hallucinations for Evaluating Image Restorationcited by 1
  34. 2026Team, Then Trim: An Assembly-Line LLM Framework for High-Quality Tabular Data Generationcited by 1
  35. 2026A Geometric Taxonomy of Hallucinations in LLMscited by 1
  36. 2026GPT-NL Public Corpus: A Permissively Licensed, Dutch-First Dataset for LLM Pre-trainingcited by 1
  37. 2026Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulationcited by 1
  38. 2026Hallucination as an Anomaly: Dynamic Intervention via Probabilistic Circuitscited by 1

50 most cited of 92 All 92 (JSON)