mlsanityData to 14 Oct 2026

Language Models with Conformal Factuality Guarantees

arXiv 2402.10978 ↗First posted 15 Feb 2024cs.LG cs.AI cs.CLcited by 131

Not on the map. Why?

Abstract (arXiv)

Guaranteeing the correctness and factuality of language model (LM) outputs is a major open problem. In this work, we propose conformal factuality, a framework that can ensure high probability correctness guarantees for LMs by connecting language modeling and conformal prediction. We observe that the correctness of an LM output is equivalent to an uncertainty quantification problem, where the uncertainty sets are defined as the entailment set of an LM's output. Using this connection, we show that conformal prediction in language models corresponds to a back-off algorithm that provides high probability correctness guarantees by progressively making LM outputs less specific (and expanding the associated uncertainty sets). This approach applies to any black-box LM and requires very few human-annotated samples. Evaluations of our approach on closed book QA (FActScore, NaturalQuestions) and reasoning tasks (MATH) show that our approach can provide 80-90% correctness guarantees while retaining the majority of the LM's original output.

Key facts. Cited by 131 papers in the map (data through 14 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

44 of 54 references.

References · 44

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2022Emergent Abilities of Large Language Modelscited by 2,512
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
32 more
  1. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  4. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  5. 2019Conformal Prediction Under Covariate Shiftcited by 519
  6. 2022Conformal prediction beyond exchangeabilitycited by 355
  7. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 327
  8. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  9. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  10. 2022Conformal Risk Controlcited by 299
  11. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  12. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  13. 2023Conformal Language Modelingcited by 170
  14. 2023Conformal Prediction With Conditional Guaranteescited by 152
  15. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  16. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  17. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  18. 2023Large Language Models for Robotics: A Surveycited by 133
  19. 2023Class-Conditional Conformal Prediction with Many Classescited by 109
  20. 2019Nested conformal prediction and quantile out-of-bag ensemble methodscited by 89
  21. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  22. 2023Alignment for Honestycited by 74
  23. 2023Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agentscited by 69
  24. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  25. 2022Training Uncertainty-Aware Classifiers with Conformalized Deep Learningcited by 64
  26. 2024Can AI Assistants Know What They Don't Know?cited by 50
  27. 2019Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answeringcited by 48
  28. 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 37
  29. 2023Conformal Nucleus Samplingcited by 33
  30. 2023Learning to Reject with a Fixed Predictor: Application to Decontextualizationcited by 32
  31. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  32. 2023Hallucination is the last thing you needcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 43; 2026: 62.

131 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  3. 2024Large language model validity via enhanced conformal prediction methodscited by 92
  4. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 76
  5. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 65
  6. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 54
  7. 2024A Closer Look at Claim Decompositioncited by 41
  8. 2024Knowledge Boundary of Large Language Models: A Surveycited by 40
  9. 2024Length Optimization in Conformal Predictioncited by 39
  10. 2024Conformal Prediction: A Data Perspectivecited by 25
  11. 2025Conformal Linguistic Calibration: Trading-off between Factuality and Specificitycited by 20
  12. 2024Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Modelscited by 19
38 more
  1. 2024Prune 'n Predict: Optimizing LLM Decision-making with Conformal Predictioncited by 18
  2. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  3. 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 17
  4. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 17
  5. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 16
  6. 2024On Subjective Uncertainty Quantification and Calibration in Natural Language Generationcited by 16
  7. 2024Core: Robust Factual Precision with Informative Sub-Claim Identificationcited by 16
  8. 2024Conformal Prediction Sets Can Cause Disparate Impactcited by 14
  9. 2025Don't lie to your friends: Learning what you know from collaborative self-playcited by 14
  10. 2025Epistemic Alignment: A Mediating Framework for User-LLM Knowledge Deliverycited by 14
  11. 2025Conformal Language Model Reasoning with Coherent Factualitycited by 14
  12. 2024Multi-group Uncertainty Quantification for Long-form Text Generationcited by 13
  13. 2025Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Modelscited by 12
  14. 2024The Role of Predictive Uncertainty and Diversity in Embodied AI and Robot Learningcited by 11
  15. 2024Conformal Risk Minimization with Variance Reductioncited by 11
  16. 2024Certifying Counterfactual Bias in LLMscited by 10
  17. 2024Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filteringcited by 10
  18. 2024Conformal Structured Predictioncited by 10
  19. 2025Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regressioncited by 10
  20. 2025Conformal Tail Risk Control for Large Language Model Alignmentcited by 8
  21. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  22. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluationcited by 8
  23. 2026Conformal Selective Acting: Anytime-Valid Risk Control for RLVR-Trained LLMscited by 8
  24. 2024FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guaranteescited by 7
  25. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  26. 2025Human-AI Collaborative Uncertainty Quantificationcited by 7
  27. 2026Entropy Alone is Insufficient for Safe Selective Prediction in LLMscited by 7
  28. 2026Adaptive Stopping for Multi-Turn LLM Reasoningcited by 7
  29. 2026PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelinescited by 7
  30. 2024LLMs can learn self-restraint through iterative self-reflectioncited by 6
  31. 2024Adapting Prediction Sets to Distribution Shifts Without Labelscited by 6
  32. 2025Document Summarization with Conformal Importance Guaranteescited by 6
  33. 2025Domain-Shift-Aware Conformal Prediction for Large Language Modelscited by 6
  34. 2026Conformal Policy Controlcited by 6
  35. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  36. 2025Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequenciescited by 5
  37. 2026Geometry-Calibrated Conformal Abstention for Language Modelscited by 5
  38. 2026Two Axes of LLM Abstention: Answer Correctness and Question Answerabilitycited by 5

50 most cited of 131 All 131 (JSON)