mlsanityData to 7 Oct 2026

Estimating Tail Risks in Language Model Output Distributions

arXiv 2604.22167 ↗First posted 24 Apr 2026cs.LG cs.AIcited by 6

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Language models are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 26 of its 52 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

26 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun

Other references · 25

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  7. 2024Phi-4 Technical Reportcited by 675
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2024A StrongREJECT for Empty Jailbreakscited by 352
  12. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
13 more
  1. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
  2. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  3. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  5. 2025Eliciting Language Model Behaviors with Investigator Agentscited by 21
  6. 2025Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)cited by 19
  7. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
  8. 2025Forecasting Rare Language Model Behaviorscited by 16
  9. 2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 15
  10. 2025Sampling-aware Adversarial Attacks Against Large Language Modelscited by 9
  11. 2026Rare Event Analysis of Large Language Modelscited by 6
  12. 2022Predictive Querying for Autoregressive Neural Sequence Modelscited by 5
  13. 2022Probabilistic Querying of Continuous-Time Event Sequencescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Predicting LLM Safety Before Release by Simulating Deploymentcited by 5
  2. 2026Estimating Rare Events in Language Models with Proper Evaluationcited by 1
  3. 2026Sound Probabilistic Safety Bounds for Large Language Modelscited by 0
  4. 2026Prioritizing Repeated LLM Evaluation for Hidden Failure Discoverycited by 0
  5. 2026Quantifying Behavioral Tails in Black-Box Language Modelscited by 0
  6. 2026Estimating great expectations under autoregressive language models with potentialscited by 0