mlsanityData to 7 Oct 2026

MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMs

arXiv 2402.11756 ↗First posted 19 Feb 2024cs.CL cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Generative Large Language Models (LLMs) are widely utilized for their excellence in various tasks. However, their tendency to produce inaccurate or misleading outputs poses a potential risk, particularly in high-stakes environments. Therefore, estimating the correctness of generative LLM outputs is an important task for enhanced reliability. Uncertainty Estimation (UE) in generative LLMs is an evolving domain, where SOTA probability-based methods commonly employ length-normalized scoring. In this work, we propose Meaning-Aware Response Scoring (MARS) as an alternative to length-normalized scoring for UE methods. MARS is a novel scoring function that considers the semantic contribution of each token in the generated sequence in the context of the question. We demonstrate that integrating MARS into UE methods results in a universal and significant improvement in UE performance. We conduct experiments using three distinct closed-book question-answering datasets across five popular pre-trained LLMs. Lastly, we validate the efficacy of MARS on a Medical QA dataset. Code can be found https://github.com/Ybakman/LLM_Uncertainity.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 25 of its 45 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

25 of 45 references.

References · 25

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2023The Falcon Series of Open Language Modelscited by 526
  10. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  12. 2020Calibration of Pre-trained Transformerscited by 307
13 more
  1. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  2. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 244
  3. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  4. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 209
  5. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
  6. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  7. 2021WebQA: Multihop and Multimodal QAcited by 109
  8. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  9. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
  10. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  11. 2022Benchmarking Bayesian Deep Learning on Diabetic Retinopathy Detection Taskscited by 49
  12. 2023Adapting Large Language Models to Domains via Reading Comprehensioncited by 48
  13. 2020Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformerscited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 23; 2026: 11.

45 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  2. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  3. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
  4. 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
  5. 2024Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluationcited by 29
  6. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  7. 2025Leveraging Uncertainty Estimation for Efficient LLM Routingcited by 21
  8. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  9. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 20
  10. 2025Robust Hallucination Detection in LLMs via Adaptive Token Selectioncited by 19
  11. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  12. 2025Reconsidering LLM Uncertainty Estimation Methods in the Wildcited by 17
33 more
  1. 2025Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorerscited by 16
  2. 2024Graph-based Confidence Calibration for Large Language Modelscited by 10
  3. 2025RAG-Check: Evaluating Multimodal Retrieval Augmented Generation Performancecited by 10
  4. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  5. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7
  6. 2024UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questionscited by 6
  7. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 6
  8. 2025Uncertainty Unveiled: Can Exposure to More In-context Examples Mitigate Uncertainty for Large Language Models?cited by 5
  9. 2025Uncertainty Quantification for Retrieval-Augmented Reasoningcited by 5
  10. 2025Probabilities Are All You Need: A Probability-Only Approach to Uncertainty Estimation in Large Language Modelscited by 5
  11. 2025TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputscited by 4
  12. 2025Towards Harmonized Uncertainty Estimation for Large Language Modelscited by 3
  13. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
  14. 2026Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQAcited by 3
  15. 2024Probing LLM Hallucination from Within: Perturbation-Driven Approach via Internal Knowledgecited by 2
  16. 2024Variability Need Not Imply Error: The Case of Adequate but Semantically Distinct Responsescited by 2
  17. 2025ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimationcited by 2
  18. 2025Confidence-Based Response Abstinence: Improving LLM Trustworthiness via Activation-Based Uncertainty Estimationcited by 2
  19. 2026Uncertainty Quantification for LLM Function-Callingcited by 2
  20. 2024Label-Confidence-Aware Uncertainty Estimation in Natural Language Generationcited by 1
  21. 2025Conformal Prediction Adaptive to Unknown Subpopulation Shiftscited by 1
  22. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learningcited by 1
  23. 2026Hallucination as Commitment Failure: Larger LLMs Misfire Despite Knowing the Answercited by 1
  24. 2026Monitoring Web Agents Without Internal Signals: Observable Trajectories and Key-Step Supervisioncited by 1
  25. 2025Invoke Interfaces Only When Needed: Adaptive Invocation for Large Language Models in Question Answeringcited by 0
  26. 2025A Systematic Analysis of Base Model Choice for Reward Modelingcited by 0
  27. 2025Un-considering Contextual Information: Assessing LLMs' Understanding of Indexical Elementscited by 0
  28. 2026Lowest Span Confidence: Zero-Shot Hallucination Detection from a Single LLM Responsecited by 0
  29. 2026FAITH: Factuality Alignment through Integrating Trustworthiness and Honestnesscited by 0
  30. 2026Gradients with Respect to Semantics Preserving Embeddings Tell the Uncertainty of Large Language Modelscited by 0
  31. 2026Uncertainty Quantification for Multimodal Retrieval Augmented Generationcited by 0
  32. 2026Confidence Composition for Multiagent Language Model Systemscited by 0
  33. 2026TRACE: Single-Pass Decoding-Trace Risk Localization for Generation Calibrationcited by 0