mlsanityData to 7 Oct 2026

Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMs

arXiv 2406.11278 ↗First posted 17 Jun 2024cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Uncertainty estimation (UE) of generative large language models (LLMs) is crucial for evaluating the reliability of generated sequences. A significant subset of UE methods utilize token probabilities to assess uncertainty, aggregating multiple token probabilities into a single UE score using a scoring function. Existing scoring functions for probability-based UE, such as length-normalized scoring and semantic contribution-based weighting, are designed to solve certain aspects of the problem but exhibit limitations, including the inability to handle biased probabilities and complex semantic dependencies between tokens. To address these issues, in this work, we propose Learnable Response Scoring (LARS) function, a novel scoring function that leverages supervised data to capture complex dependencies between tokens and probabilities, thereby producing more reliable and calibrated response scores in computing the uncertainty of LLM generations. Our comprehensive experiments across question-answering and arithmetical reasoning tasks with various datasets demonstrate that LARS significantly outperforms existing scoring functions, achieving improvements of up to 16% AUROC score.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

28 of 45 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  10. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  11. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  12. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
16 more
  1. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  2. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  3. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  4. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  5. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  6. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  7. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  8. 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 86
  9. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  10. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  11. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  12. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  13. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 45
  14. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
  15. 2023Conformal Autoregressive Generation: Beam Search with Coverage Guaranteescited by 16
  16. 2024Do LLMs Recognize me, When I is not me: Assessment of LLMs Understanding of Turkish Indexical Pronouns in Indexical Shift Contextscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 7.

23 papers.

Sort:

By area:

  1. 2025Leveraging Uncertainty Estimation for Efficient LLM Routingcited by 21
  2. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  3. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  4. 2025Reconsidering LLM Uncertainty Estimation Methods in the Wildcited by 17
  5. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  6. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 6
  7. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 5
  8. 2025TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputscited by 4
  9. 2025Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Makingcited by 4
  10. 2026Semantic Self-Distillation for Language Model Uncertaintycited by 4
  11. 2025Towards Harmonized Uncertainty Estimation for Large Language Modelscited by 3
  12. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
11 more
  1. 2026Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQAcited by 3
  2. 2024Probing LLM Hallucination from Within: Perturbation-Driven Approach via Internal Knowledgecited by 2
  3. 2025SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Modelscited by 2
  4. 2025HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answeringcited by 2
  5. 2025Probabilistic distances-based hallucination detection in LLMs with RAGcited by 1
  6. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learningcited by 1
  7. 2025Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elementscited by 0
  8. 2026Confident in a Confidence Score: Investigating the Sensitivity of Confidence Scores to Supervised Fine-Tuningcited by 0
  9. 2026Knowledge Dependency Estimation for Reliable Question Answeringcited by 0
  10. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0
  11. 2026TRACE: Single-Pass Decoding-Trace Risk Localization for Generation Calibrationcited by 0