mlsanityData to 7 Oct 2026

SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models

arXiv 2403.02509 ↗First posted 4 Mar 2024cs.CL cs.AIcited by 54

Not on the map. Why?

Abstract (arXiv)

In recent years, large language models (LLMs) have become increasingly prevalent, offering remarkable text generation capabilities. However, a pressing challenge is their tendency to make confidently wrong predictions, highlighting the critical need for uncertainty quantification (UQ) in LLMs. While previous works have mainly focused on addressing aleatoric uncertainty, the full spectrum of uncertainties, including epistemic, remains inadequately explored. Motivated by this gap, we introduce a novel UQ method, sampling with perturbation for UQ (SPUQ), designed to tackle both aleatoric and epistemic uncertainties. The method entails generating a set of perturbations for LLM inputs, sampling outputs for each perturbation, and incorporating an aggregation module that generalizes the sampling uncertainty approach for text generation tasks. Through extensive experiments on various datasets, we investigated different perturbation and aggregation techniques. Our findings show a substantial improvement in model uncertainty calibration, with a reduction in Expected Calibration Error (ECE) by 50% on average. Our findings suggest that our proposed UQ method offers promising steps toward enhancing the reliability and trustworthiness of LLMs.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 4 Mar 2024.

Builds on

22 of 31 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
10 more
  1. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  2. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  3. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  5. 2018CoQA: A Conversational Question Answering Challengecited by 674
  6. 2023How Language Model Hallucinations Can Snowballcited by 310
  7. 2022Prompting GPT-3 To Be Reliablecited by 258
  8. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  9. 2019Exploiting Epistemic Uncertainty of Anatomy Segmentation for Anomaly Detection in Retinal OCTcited by 41
  10. 2021Exploiting epistemic uncertainty of the deep learning models to generate adversarial samplescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 21; 2026: 26.

54 papers.

Sort:

By area:

  1. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  2. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 56
  3. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  4. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
  5. 2025Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)cited by 20
  6. 2024Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Modelscited by 18
  7. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  8. 2025Semantic Energy: Detecting LLM Hallucination Beyond Entropycited by 18
  9. 2025MUR: Momentum Uncertainty guided Reasoning for Large Language Modelscited by 17
  10. 2025Enhancing Hallucination Detection through Noise Injectioncited by 13
  11. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  12. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
38 more
  1. 2025Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Modelscited by 6
  2. 2025The Consistency Hypothesis in Uncertainty Quantification for Large Language Modelscited by 6
  3. 2024Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMscited by 5
  4. 2025CoRE: Enhancing Metacognition with Label-free Self-evaluation in LRMscited by 5
  5. 2025D2HScore: Reasoning-Aware Hallucination Detection via Semantic Breadth and Depth Analysis in LLMscited by 5
  6. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Modelscited by 5
  7. 2024Understanding The Effect Of Temperature On Alignment With Human Opinionscited by 4
  8. 2025Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selectioncited by 4
  9. 2025Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmissioncited by 4
  10. 2026Recurrent Confidence Chain: Temporal-Aware Uncertainty Quantification in Large Language Modelscited by 4
  11. 2026Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Methodcited by 4
  12. 2026Semantic Self-Distillation for Language Model Uncertaintycited by 4
  13. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 4
  14. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  15. 2025Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processescited by 3
  16. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agentscited by 3
  17. 2024Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responsescited by 2
  18. 2025Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillationcited by 2
  19. 2026Finding the Cracks: Improving LLMs Reasoning with Paraphrastic Probing and Consistency Verificationcited by 2
  20. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  21. 2025Exploring Membership Inference Vulnerabilities in Clinical Large Language Modelscited by 1
  22. 2025Epistemic Familiarity is Associated With Belief Stability in Large Language Modelscited by 1
  23. 2026VideoHEDGE: Entropy-Based Hallucination Detection for Video-VLMs via Semantic Clustering and Spatiotemporal Perturbationscited by 1
  24. 2026Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillationcited by 1
  25. 2026GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamicscited by 1
  26. 2026Conf-Gen: Conformal Uncertainty Quantification for Generative Modelscited by 1
  27. 2026CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distractioncited by 1
  28. 2025PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relationscited by 0
  29. 2025Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Modelscited by 0
  30. 2026From Out-of-Distribution Detection to Hallucination Detection: A Geometric Viewcited by 0
  31. 2026Directional Concentration Uncertainty: A representational approach to uncertainty quantification for generative modelscited by 0
  32. 2026Calibrated Confidence Estimation for Tabular Question Answeringcited by 0
  33. 2026Sanity Checks for Long-Form Hallucination Detectioncited by 0
  34. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  35. 2026ECUASn: A family of metrics for principled evaluation of uncertainty-augmented systemscited by 0
  36. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  37. 2026Localizing Input Uncertainty Quantification for Large Language Models via Shapley Valuescited by 0
  38. 2026FLaG: Fine-Grained Latent Grouping for Hallucination Detectioncited by 0

50 most cited of 54 All 54 (JSON)