mlsanityData to 7 Oct 2026

Conformal Prediction for Natural Language Processing: A Survey

arXiv 2405.01976 ↗First posted 3 May 2024cs.CL cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

The rapid proliferation of large language models and natural language processing (NLP) applications creates a crucial need for uncertainty quantification to mitigate risks such as hallucinations and to enhance decision-making reliability in critical applications. Conformal prediction is emerging as a theoretically sound and practically useful framework, combining flexibility with strong statistical guarantees. Its model-agnostic and distribution-free nature makes it particularly promising to address the current shortcomings of NLP systems that stem from the absence of uncertainty quantification. This paper provides a comprehensive survey of conformal prediction techniques, their guarantees, and existing applications in NLP, pointing to directions for future research and open challenges.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 86 of its 132 references that are in the map. First posted on arXiv 3 May 2024.

Builds on

86 of 132 references.

References · 86

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  10. 2016Distribution-Free Predictive Inference For Regressioncited by 657
  11. 2019Attention is not not Explanationcited by 629
  12. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
38 more
  1. 2019Conformal Prediction Under Covariate Shiftcited by 511
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  3. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  4. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  5. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  6. 2022Conformal prediction beyond exchangeabilitycited by 348
  7. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  8. 2023Explainability for Large Language Models: A Surveycited by 331
  9. 2020Calibration of Pre-trained Transformerscited by 307
  10. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  11. 2022Confident Adaptive Language Modelingcited by 297
  12. 2022Conformal Risk Controlcited by 290
  13. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  14. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  15. 2019Predictive inference with the jackknife+cited by 206
  16. 2019SPoC: Search-based Pseudocode to Codecited by 176
  17. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  18. 2023Conformal Language Modelingcited by 163
  19. 2022Adaptive Conformal Predictions for Time Seriescited by 155
  20. 2023Conformal Prediction With Conditional Guaranteescited by 146
  21. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  22. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  23. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  24. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  25. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  26. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  27. 2022Conformal Inference for Online Prediction with Arbitrary Distribution Shiftscited by 123
  28. 2019Distributional conformal predictioncited by 104
  29. 2020Mitigating Bias in Calibration Error Estimationcited by 100
  30. 2021Distribution-free uncertainty quantification for classification under label shiftcited by 99
  31. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 91
  32. 2023Conformal PID Control for Time Series Predictioncited by 85
  33. 2020Robust Validation: Confident Predictions Even When Distributions Shiftcited by 82
  34. 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
  35. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  36. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  37. 2018Exact and Robust Conformal Inference Methods for Predictive Machine Learning With Dependent Datacited by 64
  38. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 24; 2026: 24.

52 papers.

Sort:

By area:

  1. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62
  2. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  3. 2024Conformal Prediction: A Data Perspectivecited by 25
  4. 2024Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Modelscited by 18
  5. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  6. 2025Certainty in Uncertainty: Reasoning over Uncertain Knowledge Graphs with Statistical Guaranteescited by 9
  7. 2025Language Models Can Predict Their Own Behaviorcited by 8
  8. 2025Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'cited by 8
  9. 2026PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelinescited by 7
  10. 2025Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Designcited by 5
  11. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Controlcited by 5
  12. 2026Geometry-Calibrated Conformal Abstention for Language Modelscited by 5
38 more
  1. 2025How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyondcited by 4
  2. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 4
  3. 2026When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generationcited by 4
  4. 2024A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreementcited by 3
  5. 2025An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoningcited by 3
  6. 2025Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Predictioncited by 3
  7. 2025Generative AI for Autonomous Driving: A Reviewcited by 3
  8. 2025You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Controlcited by 3
  9. 2025Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answeringcited by 3
  10. 2025Statistical Management of the False Discovery Rate in Medical Instance Segmentation Based on Conformal Risk Controlcited by 2
  11. 2025Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulationcited by 2
  12. 2025Are you sure? Measuring models bias in content moderation through uncertaintycited by 2
  13. 2025Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretrainingcited by 2
  14. 2026Flow-Based Conformal Predictive Distributionscited by 2
  15. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generationcited by 2
  16. 2026Predicting Mergeability of Parameter-Efficient Fine-Tuning Updatescited by 2
  17. 2025Correctness Coverage Evaluation for Medical Multiple-Choice Question Answering Based on the Enhanced Conformal Prediction Frameworkcited by 1
  18. 2026Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domainscited by 1
  19. 2026Conformal Certification of Reasoning Trace Prefixescited by 1
  20. 2026Capability Self-Assessment in Large Language Modelscited by 1
  21. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1
  22. 2026Uncertainty-Aware Generation and Decision-Making Under Ambiguitycited by 1
  23. 2026On Optimal Data Splitting for Split Conformal Predictioncited by 1
  24. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guaranteescited by 1
  25. 2025Assumption-free fidelity bounds for hardware noise characterizationcited by 0
  26. 2025QUTCC: Quantile Uncertainty Training and Conformal Calibration for Imaging Inverse Problemscited by 0
  27. 2025Conformal Prediction and Trustworthy AIcited by 0
  28. 2025Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusalcited by 0
  29. 2025Taxonomy of Comprehensive Safety for Clinical Agentscited by 0
  30. 2025CoFineLLM: Conformal Finetuning of LLMs for Language-Instructed Robot Planningcited by 0
  31. 2026Uncertainty Quantification for Named Entity Recognition via Full-Sequence and Subsequence Conformal Predictioncited by 0
  32. 2026No Certificate, No Categorical Speech Act: A Brouwerian Assertibility Constraint for Public Reasoncited by 0
  33. 2026Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuningcited by 0
  34. 2026The Ghost Annotator: a Framework to Explore Human Label Variation in Content Moderation through Conformal Predictioncited by 0
  35. 2026Judgment-Grounded Expansion for Peer Review Generationcited by 0
  36. 2026AI YOU Town: Make Friends and Money with Your Digital Twincited by 0
  37. 2026Conformalized Large Language Models under Configuration Shiftcited by 0
  38. 2026Robust Conformal Consensus: Multi-Agent LLM-as-a-Judge Interval Evaluation with Conformal Predictioncited by 0

50 most cited of 52 All 52 (JSON)