# Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI

arXiv: https://arxiv.org/abs/2311.01463 · HTML: https://mlsanity.com/paper/2311.01463/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 26 Sep 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://mlsanity.com/paper/2305.16739/) (2023)
- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)
- [Benchmarking Large Language Models on CMExam -- A Comprehensive Chinese Medical Exam Dataset](https://mlsanity.com/paper/2306.03030/) (2023)
- [Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations](https://mlsanity.com/paper/2303.18027/) (2023)
- [An Efficient Memory-Augmented Transformer for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2210.16773/) (2022)
- [Foresight -- Generative Pretrained Transformer (GPT) for Modelling of Patient Timelines using EHRs](https://mlsanity.com/paper/2212.08072/) (2022)
- [Challenges in Domain-Specific Abstractive Summarization and How to Overcome them](https://mlsanity.com/paper/2307.00963/) (2023)

## Built on by (most cited first, 30 of 33)

- [ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence](https://mlsanity.com/paper/2404.10198/) (2024)
- [A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Models](https://mlsanity.com/paper/2405.09589/) (2024)
- [DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination](https://mlsanity.com/paper/2410.04514/) (2024)
- [JMLR: Joint Medical LLM and Retrieval Training for Enhancing Reasoning and Professional Question Answering Capability](https://mlsanity.com/paper/2402.17887/) (2024)
- [Exploring Autonomous Agents through the Lens of Large Language Models: A Review](https://mlsanity.com/paper/2404.04442/) (2024)
- [A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcare](https://mlsanity.com/paper/2502.15871/) (2025)
- [DeCoRe: Decoding by Contrasting Retrieval Heads to Mitigate Hallucinations](https://mlsanity.com/paper/2410.18860/) (2024)
- [Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization](https://mlsanity.com/paper/2407.12848/) (2024)
- [Fact-Aware Multimodal Retrieval Augmentation for Accurate Medical Radiology Report Generation](https://mlsanity.com/paper/2407.15268/) (2024)
- [How well do LLMs cite relevant medical references? An evaluation framework and analyses](https://mlsanity.com/paper/2402.02008/) (2024)
- [Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data](https://mlsanity.com/paper/2408.13833/) (2024)
- [Automatic Evaluation of Healthcare LLMs Beyond Question-Answering](https://mlsanity.com/paper/2502.06666/) (2025)
- [Correctness is not Faithfulness in RAG Attributions](https://mlsanity.com/paper/2412.18004/) (2024)
- [A Survey on Trustworthiness in Foundation Models for Medical Image Analysis](https://mlsanity.com/paper/2407.15851/) (2024)
- [Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models](https://mlsanity.com/paper/2408.13808/) (2024)
- [Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attacks](https://mlsanity.com/paper/2412.16708/) (2024)
- [MedCite: Can Language Models Generate Verifiable Text for Medicine?](https://mlsanity.com/paper/2506.06605/) (2025)
- [ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models](https://mlsanity.com/paper/2403.05266/) (2024)
- [Trustworthy Medical Imaging with Large Language Models: A Study of Hallucinations Across Modalities](https://mlsanity.com/paper/2508.07031/) (2025)
- [A Survey of Foundation Models for IoT: Taxonomy and Criteria-Based Analysis](https://mlsanity.com/paper/2506.12263/) (2025)
- [SlideBot: A Multi-Agent Framework for Generating Informative, Reliable, Multi-Modal Presentations](https://mlsanity.com/paper/2511.09804/) (2025)
- [Applications of Generative AI in Healthcare: algorithmic, ethical, legal and societal considerations](https://mlsanity.com/paper/2406.10632/) (2024)
- [Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization](https://mlsanity.com/paper/2502.19127/) (2025)
- [A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination](https://mlsanity.com/paper/2505.00008/) (2025)
- [Trustworthy Agents for Electronic Health Records through Confidence Estimation](https://mlsanity.com/paper/2508.19096/) (2025)
- [Increasing LLM response trustworthiness using voting ensembles](https://mlsanity.com/paper/2510.04048/) (2025)
- [A Checklist for Trustworthy, Safe, and User-Friendly Mental Health Chatbots](https://mlsanity.com/paper/2601.15412/) (2026)
- [Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study](https://mlsanity.com/paper/2602.14357/) (2026)
- [Effects of Different Prompts on the Quality of GPT-4 Responses to Dementia Care Questions](https://mlsanity.com/paper/2404.08674/) (2024)
- [Effects of Cross-lingual Evidence in Multilingual Medical Question Answering](https://mlsanity.com/paper/2604.20531/) (2026)

## Abstract (arXiv)

Large language models have proliferated across multiple domains in as short period of time. There is however hesitation in the medical and healthcare domain towards their adoption because of issues like factuality, coherence, and hallucinations. Give the high stakes nature of healthcare, many researchers have even cautioned against its usage until these issues are resolved. The key to the implementation and deployment of LLMs in healthcare is to make these models trustworthy, transparent (as much possible) and explainable. In this paper we describe the key elements in creating reliable, trustworthy, and unbiased models as a necessary condition for their adoption in healthcare. Specifically we focus on the quantification, validation, and mitigation of hallucinations in the context in healthcare. Lastly, we discuss how the future of LLMs in healthcare may look like.