mlsanityData to 7 Oct 2026

MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models

arXiv 2502.14302 ↗First posted 20 Feb 2025cs.CL cs.AI cs.LGcited by 33

Not on the map. Why?

Abstract (arXiv)

Advancements in Large Language Models (LLMs) and their increasing use in medical question-answering necessitate rigorous evaluation of their reliability. A critical challenge lies in hallucination, where models generate plausible yet factually incorrect outputs. In the medical domain, this poses serious risks to patient safety and clinical decision-making. To address this, we introduce MedHallu, the first benchmark specifically designed for medical hallucination detection. MedHallu comprises 10,000 high-quality question-answer pairs derived from PubMedQA, with hallucinated answers systematically generated through a controlled pipeline. Our experiments show that state-of-the-art LLMs, including GPT-4o, Llama-3.1, and the medically fine-tuned UltraMedical, struggle with this binary hallucination detection task, with the best model achieving an F1 score as low as 0.625 for detecting "hard" category hallucinations. Using bidirectional entailment clustering, we show that harder-to-detect hallucinations are semantically closer to ground truth. Through experiments, we also show incorporating domain-specific knowledge and introducing a "not sure" category as one of the answer categories improves the precision and F1 scores by up to 38% relative to baselines.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

33 of 48 references.

References · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
21 more
  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  2. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  3. 2020Recipes for building an open-domain chatbotcited by 593
  4. 2024BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domainscited by 299
  5. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  6. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  7. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  8. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  9. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  10. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  11. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 85
  12. 2024UltraMedical: Building Specialized Generalists in Biomedicinecited by 67
  13. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  14. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  15. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  16. 2019Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialoguecited by 35
  17. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  18. 2023Learn to Refuse: Making Large Language Models More Controllable and Reliable through Knowledge Scope Limitation and Refusal Mechanismcited by 33
  19. 2022Towards Improving Faithfulness in Abstractive Summarizationcited by 27
  20. 2024KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checkingcited by 26
  21. 2024Using LLMs in Software Requirements Specifications: An Empirical Evaluationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 24.

33 papers.

Sort:

By area:

  1. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  2. 2025MIRIAD: Augmenting LLMs with millions of medical query-response pairscited by 8
  3. 2026HalluHard: A Hard Multi-Turn Hallucination Benchmarkcited by 7
  4. 2025Hallucinations and Key Information Extraction in Medical Texts: A Comprehensive Assessment of Open-Source Large Language Modelscited by 4
  5. 2025MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicinecited by 4
  6. 2026CLIR-Bench: Benchmarking Multimodal Question Answering over Irregular Clinical Time Seriescited by 4
  7. 2025Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logscited by 3
  8. 2025MedHal: a Synthetic Dataset for Medical Hallucination Detectioncited by 2
  9. 2025Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systemscited by 2
  10. 2025Multi-agent Self-triage System with Medical Flowchartscited by 2
  11. 2026Hallucination Detection via Activations of Open-Weight Proxy Analyzerscited by 2
  12. 2026HalluScore: Large Language Model Hallucination Question Answering Benchmarkcited by 2
21 more
  1. 2026BalanceRAG: Joint Risk Calibration for Cascaded Retrieval-Augmented Generationcited by 2
  2. 2025MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Datacited by 1
  3. 2026Toward Reliable, Safe, and Secure LLMs for Scientific Applicationscited by 1
  4. 2026FinReflectKG -- HalluBench: GraphRAG Hallucination Benchmark for Financial Question Answering Systemscited by 1
  5. 2026The Path to Self-Evolving Clinical Systems: Scaling Medical Agents from Assistance to Autonomycited by 1
  6. 2026Unified Hallucination Fuzzing for Multimodal Large Language Modelscited by 1
  7. 2026Logi-PAR: Logic-Infused Patient Activity Recognition via Differentiable Rulecited by 0
  8. 2026A Multi-Stage Validation Framework for Trustworthy Large-scale Clinical Information Extraction using Large Language Modelscited by 0
  9. 2026BLUEmed: Retrieval-Augmented Multi-Agent Debate for Clinical Error Detectioncited by 0
  10. 2026CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMscited by 0
  11. 2026Self-Attention as Transport: Limits of Symmetric Spectral Diagnosticscited by 0
  12. 2026Graph Alignment Topology as an Inductive Bias for Grounding Detectioncited by 0
  13. 2026Towards Lightweight Reliability: Using Soft Prompts for Hallucination Mitigation in Large Language Modelscited by 0
  14. 2026Med-HEAL: Analyzing and Mitigating Hallucinations in Medical LLMs with Hallucination-Aware In-Context Learningcited by 0
  15. 2026ERRORQUAKE: Heavy-Tailed Error Severity Distributions in Open-Weight Large Language Modelscited by 0
  16. 2026MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Modelscited by 0
  17. 2026Toward Workflow-Aware Benchmarking for Healthcare NLP Agentscited by 0
  18. 2026Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakagecited by 0
  19. 2026Route, Don't Fix: Regime-Dependent Decoding Correction and a Trajectory-Gated Router for Reliable Clinical LLM Answer Selectioncited by 0
  20. 2026PROVE: Proof-guided Regime-aware Operator Verification for Hallucination Detection in Medical Visual Question Answeringcited by 0
  21. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0