mlsanityData to 7 Oct 2026

Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation

arXiv 2406.01806 ↗First posted 3 Jun 2024cs.CL cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

The advent of large language models (LLMs) has dramatically advanced the state-of-the-art in numerous natural language generation tasks. For LLMs to be applied reliably, it is essential to have an accurate measure of their confidence. Currently, the most commonly used confidence score function is the likelihood of the generated sequence, which, however, conflates semantic and syntactic components. For instance, in question-answering (QA) tasks, an awkward phrasing of the correct answer might result in a lower probability prediction. Additionally, different tokens should be weighted differently depending on the context. In this work, we propose enhancing the predicted sequence probability by assigning different weights to various tokens using attention values elicited from the base LLM. By employing a validation set, we can identify the relevant attention heads, thereby significantly improving the reliability of the vanilla sequence probability confidence measure. We refer to this new score as the Contextualized Sequence Likelihood (CSL). CSL is easy to implement, fast to compute, and offers considerable potential for further improvement with task-specific prompts. Across several QA datasets and a diverse array of LLMs, CSL has demonstrated significantly higher reliability than state-of-the-art baselines in predicting generation quality, as measured by the AUROC or AUARC.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 32 of its 58 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

32 of 58 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  6. 2017Selective Classification for Deep Neural Networkscited by 799
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2018CoQA: A Conversational Question Answering Challengecited by 674
  9. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  10. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  12. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
20 more
  1. 2020Are we done with ImageNet?cited by 329
  2. 2020Calibration of Pre-trained Transformerscited by 307
  3. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  4. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  5. 2020Selective Question Answering under Domain Shiftcited by 228
  6. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  7. 2018To Trust Or Not To Trust A Classifiercited by 205
  8. 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
  9. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  10. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  11. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  12. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  13. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  14. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  15. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 48
  16. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 40
  17. 2023ReLM: Leveraging Language Models for Enhanced Chemical Reaction Predictioncited by 24
  18. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20
  19. 2020Towards Improving Selective Prediction Ability of NLP Systemscited by 19
  20. 2021SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Makingcited by 36
  2. 2025Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Headscited by 14
  3. 2025Language Model Uncertainty Quantification with Attention Chaincited by 13
  4. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  5. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  6. 2025On LLMs' Internal Representation of Code Correctnesscited by 5
  7. 2024Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responsescited by 2
  8. 2025Confident RAG: Enhancing the Performance of LLMs for Mathematics Question Answering through Multi-Embedding and Confidence Scoringcited by 2
  9. 2025Efficient semantic uncertainty quantification in language models via diversity-steered samplingcited by 2
  10. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  11. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  12. 2026Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revisioncited by 0
4 more
  1. 2026ECUASn: A family of metrics for principled evaluation of uncertainty-augmented systemscited by 0
  2. 2026Evaluating the Relevance of Uncertainty Estimators for LLM Hallucinationcited by 0
  3. 2026Hierarchy-Aware Supervised Uncertainty Estimation for Black-box LLM Taxonomic Reasoningcited by 0
  4. 2026GUT: Quantifying and Optimizing the Reasoning Uncertainty of LLMs via Graph Complexitycited by 0