mlsanityData to 7 Oct 2026

Non-Exchangeable Conformal Language Generation with Nearest Neighbors

arXiv 2402.00707 ↗First posted 1 Feb 2024cs.CL cs.AI cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Quantifying uncertainty in automatically generated text is important for letting humans check potential hallucinations and making systems more reliable. Conformal prediction is an attractive framework to provide predictions imbued with statistical guarantees, however, its application to text generation is challenging since any i.i.d. assumptions are not realistic. In this paper, we bridge this gap by leveraging recent results on non-exchangeable conformal prediction, which still ensures bounds on coverage. The result, non-exchangeable conformal nucleus sampling, is a novel extension of the conformal prediction framework to generation based on nearest neighbors. Our method can be used post-hoc for an arbitrary model without extra training and supplies token-level, calibrated prediction sets equipped with statistical guarantees. Experiments in machine translation and language modeling show encouraging results in generation quality. By also producing tighter prediction sets with good coverage, we thus give a more theoretically principled way to perform sampling with conformal guarantees.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 55 of its 98 references that are in the map. First posted on arXiv 1 Feb 2024.

Builds on

55 of 98 references.

References · 55

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  12. 2018Hierarchical Neural Story Generationcited by 1,182
38 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  3. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  5. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  6. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  7. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  8. 2019Conformal Prediction Under Covariate Shiftcited by 511
  9. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  10. 2020Classification with Valid and Adaptive Coveragecited by 390
  11. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  12. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  13. 2020Nearest Neighbor Machine Translationcited by 209
  14. 2020Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translationcited by 137
  15. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  16. 2018Learning to Write with Cooperative Discriminatorscited by 119
  17. 2021Localized Conformal Prediction: A Generalized Inference Framework for Conformal Predictioncited by 112
  18. 2021Efficient Nearest Neighbor Language Modelscited by 93
  19. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  20. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  21. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  22. 2021Adaptive Nearest Neighbor Machine Translationcited by 58
  23. 2023Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translationcited by 57
  24. 2021Few-shot Conformal Prediction with Auxiliary Taskscited by 53
  25. 2022Quality-Aware Decoding for Neural Machine Translationcited by 49
  26. 2019Energy Usage Reports: Environmental awareness as part of algorithmic accountabilitycited by 43
  27. 2021Fast Nearest Neighbor Machine Translationcited by 37
  28. 2021Uncertainty-Aware Machine Translation Evaluationcited by 35
  29. 2017An optimal transportation approach for assessing almost stochastic ordercited by 33
  30. 2023Text-CRS: A Generalized Certified Robustness Framework against Textual Adversarial Attackscited by 32
  31. 2020Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformerscited by 31
  32. 2023Why do Nearest Neighbor Language Models Work?cited by 29
  33. 2022Conformal Prediction Sets with Limited False Positivescited by 28
  34. 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
  35. 2021Conformal prediction for text infilling and part-of-speech predictioncited by 22
  36. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  37. 2023Conformal Autoregressive Generation: Beam Search with Coverage Guaranteescited by 16
  38. 2022Conformal Prediction Intervals with Temporal Dependencecited by 9

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 4; 2026: 8.

21 papers.

Sort:

By area:

  1. 2024Language Models with Conformal Factuality Guaranteescited by 128
  2. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  3. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  4. 2024Conformal Prediction: A Data Perspectivecited by 25
  5. 2024Prune 'n Predict: Optimizing LLM Decision-making with Conformal Predictioncited by 15
  6. 2024Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filteringcited by 10
  7. 2026Entropy Alone is Insufficient for Safe Selective Prediction in LLMscited by 7
  8. 2025Conformal Prediction Beyond the Seen: A Missing Mass Perspective for Uncertainty Quantification in Generative Modelscited by 6
  9. 2025Domain-Shift-Aware Conformal Prediction for Large Language Modelscited by 6
  10. 2025Statistical Methods in Generative AIcited by 4
  11. 2024Uncertainty Quantification on Graph Learning: A Surveycited by 3
  12. 2024Calibrated Large Language Models for Binary Question Answeringcited by 3
9 more
  1. 2024A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreementcited by 3
  2. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  3. 2026Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Predictioncited by 2
  4. 2026Conformal Certification of Reasoning Trace Prefixescited by 1
  5. 2026Strategic Decision Support for AI Agentscited by 1
  6. 2026Directional Concentration Uncertainty: A representational approach to uncertainty quantification for generative modelscited by 0
  7. 2026ConfHit: Conformal Generative Design with Oracle Free Guaranteescited by 0
  8. 2026Uncertainty quantification for trustworthy deep learning: Methods and measurescited by 0
  9. 2026Conformalized Large Language Models under Configuration Shiftcited by 0