mlsanityData to 7 Oct 2026

Sound Probabilistic Safety Bounds for Large Language Models

arXiv 2607.20286 ↗First posted 22 Jul 2026cs.CL cs.AIcited by 0

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

We propose a novel framework for computing rigorous bounds on the probability that a large language model (LLM) generates harmful output to a given prompt. We study a new application of the Clopper-Pearson confidence intervals to obtain probably approximately correct (PAC) bounds for this problem. As our main technical contribution, we propose an algorithm that leverages features in the latent space to prioritize exploring branches in the auto-regressive generation tree that are more likely to produce harmful outputs. Our approach in particular enables the efficient computation of useful lower bounds, even in scenarios where the true harm probability is extremely small, and crucially, the obtained lower bounds are sound, i.e., formally proven to be less than the actual harmfulness probability: our experimental results demonstrate the effectiveness of our method by computing non-trivial lower bounds on state-of-the-art LLMs. This study newly enables the evaluation and statistical certification of LLMs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 25 of its 35 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

25 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Estimating Tail Risks in Language Model Output Distributions24 Apr
  2. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2022Red Teaming Language Models with Language Modelscited by 983
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2024Phi-4 Technical Reportcited by 675
  9. 2019Certified Robustness to Adversarial Word Substitutionscited by 164
  10. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  11. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  12. 2020Robustness Verification for Transformerscited by 76
11 more
  1. 2020Verification of Deep Convolutional Neural Networks Using ImageStarscited by 75
  2. 2021Towards Robustness Against Natural Language Word Substitutionscited by 68
  3. 2018A Statistical Approach to Assessing Neural Network Robustnesscited by 54
  4. 2025A Survey on Post-training of Large Language Modelscited by 53
  5. 2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocolscited by 28
  6. 2021Automated Verification and Synthesis of Stochastic Hybrid Systems: A Surveycited by 20
  7. 2022BayesFormer: Transformer with Uncertainty Estimationcited by 19
  8. 2025Forecasting Rare Language Model Behaviorscited by 16
  9. 2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 15
  10. 2022Certified Robustness to Word Substitution Ranking Attack for Neural Ranking Modelscited by 11
  11. 2023Uncertainty Estimation of Transformers' Predictions via Topological Analysis of the Attention Matricescited by 8

Built on by

Not cited yet.