# Conformal Prediction for Natural Language Processing: A Survey

arXiv: https://arxiv.org/abs/2405.01976 · HTML: https://mlsanity.com/paper/2405.01976/

## Key facts

- Cited by 53 papers in the map (data through 14 Oct 2026).
- Builds on 86 of its 132 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 30 of 86)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Distribution-Free Predictive Inference For Regression](https://mlsanity.com/paper/1604.04173/) (2016)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Adaptive Conformal Inference Under Distribution Shift](https://mlsanity.com/paper/2106.00170/) (2021)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Confident Adaptive Language Modeling](https://mlsanity.com/paper/2207.07061/) (2022)
- [Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners](https://mlsanity.com/paper/2307.01928/) (2023)
- [Conformal Risk Control](https://mlsanity.com/paper/2208.02814/) (2022)
- [Distribution-Free, Risk-Controlling Prediction Sets](https://mlsanity.com/paper/2101.02703/) (2021)
- [Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control](https://mlsanity.com/paper/2110.01052/) (2021)
- [Predictive inference with the jackknife+](https://mlsanity.com/paper/1905.02928/) (2019)
- [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://mlsanity.com/paper/2308.08493/) (2023)
- [SPoC: Search-based Pseudocode to Code](https://mlsanity.com/paper/1906.04908/) (2019)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)

## Built on by (most cited first, 30 of 53)

- [ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees](https://mlsanity.com/paper/2407.00499/) (2024)
- [Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction](https://mlsanity.com/paper/2509.18658/) (2025)
- [Conformal Prediction: A Data Perspective](https://mlsanity.com/paper/2410.06494/) (2024)
- [Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models](https://mlsanity.com/paper/2410.08174/) (2024)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Certainty in Uncertainty: Reasoning over Uncertain Knowledge Graphs with Statistical Guarantees](https://mlsanity.com/paper/2510.24754/) (2025)
- [Language Models Can Predict Their Own Behavior](https://mlsanity.com/paper/2502.13329/) (2025)
- [Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly 'Know When They Do Not Know'](https://mlsanity.com/paper/2505.04950/) (2025)
- [LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems](https://mlsanity.com/paper/2512.01556/) (2025)
- [PASC: Pipeline-Aware Conformal Prediction with Joint Coverage Guarantees for Multi-Stage NLP and LLM Pipelines](https://mlsanity.com/paper/2605.18812/) (2026)
- [Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design](https://mlsanity.com/paper/2509.12527/) (2025)
- [ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control](https://mlsanity.com/paper/2604.16677/) (2026)
- [Geometry-Calibrated Conformal Abstention for Language Models](https://mlsanity.com/paper/2604.27914/) (2026)
- [How to Enable Effective Cooperation Between Humans and NLP Models: A Survey of Principles, Formalizations, and Beyond](https://mlsanity.com/paper/2501.05714/) (2025)
- [When Can Conformal Risk Control Certify LLM Outputs? Bounds, Impossibility, and Adaptation for Structured Generation](https://mlsanity.com/paper/2606.29054/) (2026)
- [A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement](https://mlsanity.com/paper/2411.04090/) (2024)
- [An Empirical Study of Conformal Prediction in LLM with ASP Scaffolds for Robust Reasoning](https://mlsanity.com/paper/2503.05439/) (2025)
- [Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction](https://mlsanity.com/paper/2504.17671/) (2025)
- [Generative AI for Autonomous Driving: A Review](https://mlsanity.com/paper/2505.15863/) (2025)
- [Robust Instant Policy: Leveraging Student's t-Regression Model for Robust In-context Imitation Learning of Robot Manipulation](https://mlsanity.com/paper/2506.15157/) (2025)
- [You Don't Bring Me Flowers: Mitigating Unwanted Recommendations Through Conformal Risk Control](https://mlsanity.com/paper/2507.16829/) (2025)
- [Quantifying Uncertainty in Natural Language Explanations of Large Language Models for Question Answering](https://mlsanity.com/paper/2509.15403/) (2025)
- [Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation](https://mlsanity.com/paper/2604.19444/) (2026)
- [Statistical Management of the False Discovery Rate in Medical Instance Segmentation Based on Conformal Risk Control](https://mlsanity.com/paper/2504.04482/) (2025)
- [Are you sure? Measuring models bias in content moderation through uncertainty](https://mlsanity.com/paper/2509.22699/) (2025)
- [Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining](https://mlsanity.com/paper/2510.22931/) (2025)
- [Flow-Based Conformal Predictive Distributions](https://mlsanity.com/paper/2602.07633/) (2026)
- [Conformal Certification of Reasoning Trace Prefixes](https://mlsanity.com/paper/2605.30085/) (2026)
- [From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation](https://mlsanity.com/paper/2606.13221/) (2026)
- [Predicting Mergeability of Parameter-Efficient Fine-Tuning Updates](https://mlsanity.com/paper/2606.19549/) (2026)

## Abstract (arXiv)

The rapid proliferation of large language models and natural language processing (NLP) applications creates a crucial need for uncertainty quantification to mitigate risks such as hallucinations and to enhance decision-making reliability in critical applications. Conformal prediction is emerging as a theoretically sound and practically useful framework, combining flexibility with strong statistical guarantees. Its model-agnostic and distribution-free nature makes it particularly promising to address the current shortcomings of NLP systems that stem from the absence of uncertainty quantification. This paper provides a comprehensive survey of conformal prediction techniques, their guarantees, and existing applications in NLP, pointing to directions for future research and open challenges.