mlsanityData to 14 Oct 2026

Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design

arXiv 2509.12527 ↗First posted 16 Sep 2025cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Large language models often produce confident but incorrect outputs, creating a critical need for reliable uncertainty quantification with formal abstention guarantees. We introduce information-lift certificates that compare model probabilities to a skeleton baseline, accumulating evidence through sub-gamma PAC-Bayes bounds that remain valid under heavy-tailed distributions where standard concentration inequalities fail. On eight diverse datasets, our method achieves 77.0% coverage at 2% risk, outperforming recent baselines by 10.0 percentage points on average. In high-stakes scenarios, we block 96% of critical errors compared to 18-31% for entropy-based methods. While our frequency-based certification does not guarantee severity-weighted safety and depends on skeleton quality, performance degrades gracefully under distributional shifts, making the approach practical for real-world deployment.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 16 Sep 2025.

Builds on

42 of 73 references.

References · 42

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2023Mistral 7Bcited by 3,164
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
30 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  3. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  4. 2017Selective Classification for Deep Neural Networkscited by 852
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  7. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  8. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  9. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  10. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 652
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  12. 2019Conformal Prediction Under Covariate Shiftcited by 519
  13. 2020Classification with Valid and Adaptive Coveragecited by 396
  14. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  15. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  16. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
  17. 2023Conformal Language Modelingcited by 170
  18. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  19. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 105
  20. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 65
  21. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 53
  22. 2020PAC-Bayes unleashed: generalisation bounds with unbounded lossescited by 43
  23. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  24. 2020Selective Classification via One-Sided Predictioncited by 38
  25. 2022Anytime-valid off-policy inference for contextual banditscited by 37
  26. 2022Calibrated Selective Classificationcited by 33
  27. 2023Non-Exchangeable Conformal Risk Controlcited by 29
  28. 2024Active, anytime-valid risk controlling prediction setscited by 14
  29. 2024Selective Classification Under Distribution Shiftscited by 12
  30. 2023Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Anytime-Valid Answer Sufficiency Certificates for LLM Generation via Sequential Information Liftcited by 3
  2. 2026CalPro: Prior-Aware Evidential--Conformal Prediction with Structure-Aware Guarantees for Protein Structurescited by 0
  3. 2026Calibrated Adaptation: Bayesian Stiefel Manifold Priors for Reliable Parameter-Efficient Fine-Tuningcited by 0
  4. 2026Continual Calibration: Coverage Can Collapse Before Accuracy in Lifelong LLM Fine-Tuningcited by 0
  5. 2026Private Anytime Selective-Risk Certification for Federated Retrieval-Augmented Generation: Guarantees and Empirical Limitscited by 0