mlsanityData to 7 Oct 2026

LM-Polygraph: Uncertainty Estimation for Language Models

arXiv 2311.07383 ↗First posted 13 Nov 2023cs.CL cs.LGcited by 130

Not on the map. Why?

Abstract (arXiv)

Recent advancements in the capabilities of large language models (LLMs) have paved the way for a myriad of groundbreaking applications in various fields. However, a significant challenge arises as these models often "hallucinate", i.e., fabricate facts without providing users an apparent means to discern the veracity of their statements. Uncertainty estimation (UE) methods are one path to safer, more responsible, and more effective use of LLMs. However, to date, research on UE methods for LLMs has been focused primarily on theoretical rather than engineering contributions. In this work, we tackle this issue by introducing LM-Polygraph, a framework with implementations of a battery of state-of-the-art UE methods for LLMs in text generation tasks, with unified program interfaces in Python. Additionally, it introduces an extendable benchmark for consistent evaluation of UE techniques by researchers, and a demo web application that enriches the standard chat dialog with confidence scores, empowering end-users to discern unreliable responses. LM-Polygraph is compatible with the most recent LLMs, including BLOOMz, LLaMA-2, ChatGPT, and GPT-4, and is designed to support future releases of similarly-styled LMs.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

22 of 46 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2018CoQA: A Conversational Question Answering Challengecited by 674
  12. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
10 more
  1. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  2. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  3. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  4. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  5. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  6. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58
  7. 2023Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspectioncited by 44
  8. 2022Nonparametric Uncertainty Quantification for Single Deterministic Neural Networkcited by 36
  9. 2015Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systemscited by 34
  10. 2019Mitigating Uncertainty in Document Classificationcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 30; 2025: 52; 2026: 47.

130 papers.

Sort:

By area:

  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
  3. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  4. 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
  5. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  6. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  7. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  8. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 57
  9. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 54
  10. 2024Uncertainty Quantification for In-Context Learning of Large Language Modelscited by 45
  11. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  12. 2024DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Modelscited by 34
38 more
  1. 2025Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Homecited by 33
  2. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  3. 2024Towards Uncertainty-Aware Language Agentcited by 29
  4. 2024Do LLMs estimate uncertainty well in instruction-following?cited by 26
  5. 2025Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similaritycited by 26
  6. 2025The Illusion of Progress: Re-evaluating Hallucination Detection in LLMscited by 22
  7. 2024A Probabilistic Framework for LLM Hallucination Detection via Belief Tree Propagationcited by 21
  8. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  9. 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 20
  10. 2024LoGU: Long-form Generation with Uncertainty Expressionscited by 19
  11. 2024Accounting for Sycophancy in Language Model Uncertainty Estimationcited by 18
  12. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  13. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  14. 2024LRP4RAG: Detecting Hallucinations in Retrieval-Augmented Generation via Layer-wise Relevance Propagationcited by 17
  15. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  16. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 15
  17. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  18. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  19. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 13
  20. 2024Multi-group Uncertainty Quantification for Long-form Text Generationcited by 13
  21. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safetycited by 12
  22. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  23. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  24. 2024Graph-based Confidence Calibration for Large Language Modelscited by 10
  25. 2025Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoningcited by 10
  26. 2024Enabling Weak LLMs to Judge Response Reliability via Meta Rankingcited by 9
  27. 2024Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Modelscited by 9
  28. 2024Understanding the Impact of Confidence in Retrieval Augmented Generation: A Case Study in the Medical Domaincited by 9
  29. 2025Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Taskscited by 9
  30. 2025UQLM: A Python Package for Uncertainty Quantification in Large Language Modelscited by 9
  31. 2025PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chainscited by 9
  32. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  33. 2025Uncertainty Quantification in Retrieval Augmented Question Answeringcited by 8
  34. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  35. 2026AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systemscited by 8
  36. 2024Crayon: Customized On-Device LLM via Instant Adapter Blending and Edge-Server Hybrid Inferencecited by 7
  37. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  38. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7

50 most cited of 130 All 130 (JSON)