mlsanityData to 14 Oct 2026

LUQ: Long-text Uncertainty Quantification for LLMs

arXiv 2403.20279 ↗First posted 29 Mar 2024cs.CLcited by 81

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capability in a variety of NLP tasks. However, LLMs are also prone to generate nonfactual content. Uncertainty Quantification (UQ) is pivotal in enhancing our understanding of a model's confidence on its generation, thereby aiding in the mitigation of nonfactual outputs. Existing research on UQ predominantly targets short text generation, typically yielding brief, word-limited responses. However, real-world applications frequently necessitate much longer responses. Our study first highlights the limitations of current UQ methods in handling long text generation. We then introduce textscLuq and its two variations, a series of novel sampling-based UQ approaches specifically designed for long text. Our findings reveal that textscLuq outperforms existing baseline methods in correlating with the model's factuality scores (negative coefficient of -0.85 observed for Gemini Pro). To further improve the factuality of LLM responses, we propose textscLuq-Ensemble, a method that ensembles responses from multiple models and selects the response with the lowest uncertainty. The ensembling method greatly improves the response factuality upon the best standalone LLM.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

35 of 61 references.

References · 35

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  8. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  11. 2017Selective Classification for Deep Neural Networkscited by 852
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
23 more
  1. 2019ELI5: Long Form Question Answeringcited by 575
  2. 2021A Survey on Automated Fact-Checkingcited by 378
  3. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  4. 2023Do Large Language Models Know What They Don't Know?cited by 243
  5. 2020Selective Question Answering under Domain Shiftcited by 238
  6. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  7. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  8. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  9. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  10. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  11. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 122
  12. 2018Correcting Length Bias in Neural Machine Translationcited by 120
  13. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  14. 2020Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentationcited by 61
  15. 2020Learning Loss for Test-Time Augmentationcited by 56
  16. 2020Gradients as a Measure of Uncertainty in Neural Networkscited by 55
  17. 2018Classification Uncertainty of Deep Neural Networks Based on Gradient Informationcited by 43
  18. 2024Towards Uncertainty-Aware Language Agentcited by 32
  19. 2020Estimating Model Uncertainty of Neural Networks in Sparse Information Formcited by 31
  20. 2024Combining Confidence Elicitation and Sample-based Methods for Uncertainty Quantification in Misinformation Mitigationcited by 19
  21. 2023Uncertainty-aware Language Modeling for Selective Question Answeringcited by 16
  22. 2024Do We Need Language-Specific Fact-Checking Models? The Case of Chinesecited by 15
  23. 2023Large Language Models are Miscalibrated In-Context Learnerscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 32; 2026: 31.

81 papers.

Sort:

By area:

  1. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  3. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  4. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
  5. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 62
  6. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  7. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 41
  8. 2024Conformity in Large Language Modelscited by 34
  9. 2025Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similaritycited by 28
  10. 2024Atomic Calibration of LLMs in Long-Form Generationscited by 24
  11. 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 21
  12. 2024Can Large Language Models Generate High-quality Patent Claims?cited by 19
38 more
  1. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  2. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 18
  3. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversitycited by 18
  4. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  5. 2025Reconsidering LLM Uncertainty Estimation Methods in the Wildcited by 17
  6. 2025Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Headscited by 16
  7. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  8. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 14
  9. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  10. 2025When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQAcited by 14
  11. 2024Uncertainty Quantification in Large Language Models Through Convex Hull Analysiscited by 13
  12. 2025Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applicationscited by 13
  13. 2025Uncertainty Distillation: Teaching Language Models to Express Semantic Confidencecited by 13
  14. 2026Confidence Estimation for LLMs in Multi-turn Interactionscited by 12
  15. 2024Graph-based Confidence Calibration for Large Language Modelscited by 10
  16. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 10
  17. 2024Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Modelscited by 9
  18. 2024Integrative Decoding: Improve Factuality via Implicit Self-consistencycited by 8
  19. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  20. 2025Position: Towards a Responsible LLM-empowered Multi-Agent Systemscited by 8
  21. 2025Uncertainty Quantification in Retrieval Augmented Question Answeringcited by 8
  22. 2025Estimating LLM Consistency: A User Baseline vs Surrogate Metricscited by 8
  23. 2025Stop Reducing Responsibility in LLM-Powered Multi-Agent Systems to Local Alignmentcited by 7
  24. 2026Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimizationcited by 7
  25. 2025Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Modelscited by 6
  26. 2025Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantificationcited by 6
  27. 2024Enhancing Trust in Large Language Models via Uncertainty-Calibrated Fine-Tuningcited by 5
  28. 2024Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generationcited by 5
  29. 2025Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?cited by 5
  30. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  31. 2026Recurrent Confidence Chain: Temporal-Aware Uncertainty Quantification in Large Language Modelscited by 5
  32. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 4
  33. 2025TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputscited by 4
  34. 2026Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Studycited by 4
  35. 2026Code Is More Than Text: Uncertainty Estimation for Code Generationcited by 4
  36. 2025Modeling Subjectivity in Cognitive Appraisal with Language Modelscited by 3
  37. 2026Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QAcited by 3
  38. 2026Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecastingcited by 3

50 most cited of 81 All 81 (JSON)