mlsanityData to 13 Oct 2026

Conformal Autoregressive Generation: Beam Search with Coverage Guarantees

arXiv 2309.03797 ↗First posted 7 Sep 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We introduce two new extensions to the beam search algorithm based on conformal predictions (CP) to produce sets of sequences with theoretical coverage guarantees. The first method is very simple and proposes dynamically-sized subsets of beam search results but, unlike typical CP procedures, has an upper bound on the achievable guarantee depending on a post-hoc calibration measure. Our second algorithm introduces the conformal set prediction procedure as part of the decoding process, producing a variable beam width which adapts to the current uncertainty. While more complex, this procedure can achieve coverage guarantees selected a priori. We provide marginal coverage bounds for each method, and evaluate them empirically on a selection of tasks drawing from natural language processing and chemistry.

Key facts. Cited by 16 papers in the map (data through 13 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

27 of 45 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,106
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2017On Calibration of Modern Neural Networkscited by 4,902
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,893
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,823
  7. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 810
  8. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 786
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 440
  10. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  11. 2020Calibration of Pre-trained Transformerscited by 308
  12. 2022Confident Adaptive Language Modelingcited by 302
15 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 302
  2. 2022Conformal Risk Controlcited by 293
  3. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 289
  4. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 263
  5. 2018Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Predictioncited by 225
  6. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  7. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  8. 2023Conformal Language Modelingcited by 167
  9. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  10. 2017Predicting Organic Reaction Outcomes with Weisfeiler-Lehman Networkcited by 102
  11. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  12. 2022Regression Transformer: Concurrent sequence regression and generation for molecular language modelingcited by 49
  13. 2022Uncertainty Estimation for Language Reward Modelscited by 35
  14. 2023Conformal Nucleus Samplingcited by 32
  15. 2021Conformal prediction for text infilling and part-of-speech predictioncited by 22

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 2; 2026: 5.

16 papers.

Sort:

By area:

  1. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 126
  2. 2024Multicalibration for Confidence Scoring in LLMscited by 45
  3. 2024Do Not Design, Learn: A Trainable Scoring Function for Uncertainty Estimation in Generative LLMscited by 23
  4. 2024Non-Exchangeable Conformal Language Generation with Nearest Neighborscited by 21
  5. 2024Trustworthy Classification through Rank-Based Conformal Prediction Setscited by 20
  6. 2023Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Controlcited by 19
  7. 2024Prune 'n Predict: Optimizing LLM Decision-making with Conformal Predictioncited by 17
  8. 2026Knowing When to Abstain: Medical LLMs Under Clinical Uncertaintycited by 15
  9. 2024Certifying Counterfactual Bias in LLMscited by 9
  10. 2025Statistical Methods in Generative AIcited by 4
  11. 2024Conditional Shift-Robust Conformal Prediction for Graph Neural Networkcited by 1
  12. 2026Conformalized Large Language Models under Configuration Shiftcited by 1
4 more
  1. 2025CoVeR: Conformal Calibration for Versatile and Reliable Autoregressive Next-Token Predictioncited by 0
  2. 2026PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Searchcited by 0
  3. 2026Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inferencecited by 0
  4. 2026The Exceedance Design Effect: Effective Sample Size for Thresholds under Clusteringcited by 0