mlsanityData to 14 Oct 2026

Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned Models

arXiv 2409.02976 ↗First posted 4 Sep 2024cs.LG cs.AI cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Uncertainty estimation is a necessary component when implementing AI in high-risk settings, such as autonomous cars, medicine, or insurances. Large Language Models (LLMs) have seen a surge in popularity in recent years, but they are subject to hallucinations, which may cause serious harm in high-risk settings. Despite their success, LLMs are expensive to train and run: they need a large amount of computations and memory, preventing the use of ensembling methods in practice. In this work, we present a novel method that allows for fast and memory-friendly training of LLM ensembles. We show that the resulting ensembles can detect hallucinations and are a viable approach in practice as only one GPU is needed for training and inference.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

31 of 44 references.

References · 31

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
19 more
  1. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  2. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  3. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
  4. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  5. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  6. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  7. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  8. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  9. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  10. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  11. 2023Complex Claim Verification with Evidence Retrieved in the Wildcited by 110
  12. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  13. 2018Uncertainty in Neural Networks: Approximately Bayesian Ensemblingcited by 94
  14. 2023LoRA ensembles for large language model fine-tuningcited by 67
  15. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 56
  16. 2022Uncertainty Estimation for Language Reward Modelscited by 37
  17. 2024Uncertainty quantification in fine-tuned LLMs using LoRA ensemblescited by 37
  18. 2023Fast and Accurate Factual Inconsistency Detection Over Long Documentscited by 23
  19. 2023Retrieving Supporting Evidence for LLMs Generated Answerscited by 21

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 11.

15 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  2. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 55
  3. 2025Hallucination Detection in LLMs Using Spectral Features of Attention Mapscited by 43
  4. 2025The Illusion of Progress: Re-evaluating Hallucination Detection in LLMscited by 22
  5. 2025Uncertainty Distillation: Teaching Language Models to Express Semantic Confidencecited by 13
  6. 2025HalluciNot: Hallucination Detection Through Context and Common Knowledge Verificationcited by 13
  7. 2024DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Modelscited by 11
  8. 2025Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Taskscited by 10
  9. 2025HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Modelscited by 4
  10. 2024HARP: Hesitation-Aware Reframing in Transformer Inference Passcited by 3
  11. 2025REFIND at SemEval-2025 Task 3: Retrieval-Augmented Factuality Hallucination Detection in Large Language Modelscited by 3
  12. 2025Semantic uncertainty in advanced decoding methods for LLM generationcited by 3
3 more
  1. 2025Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddingscited by 3
  2. 2025Revisiting Hallucination Detection with Effective Rank-based Uncertaintycited by 3
  3. 2025Scalable Bayesian Monte Carlo: fast uncertainty estimation beyond deep ensemblescited by 1