# Estimating Tail Risks in Language Model Output Distributions

arXiv: https://arxiv.org/abs/2604.22167 · HTML: https://mlsanity.com/paper/2604.22167/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs](https://mlsanity.com/paper/2410.05295/) (2024)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Natural Emergent Misalignment from Reward Hacking in Production RL](https://mlsanity.com/paper/2511.18397/) (2025)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Eliciting Language Model Behaviors with Investigator Agents](https://mlsanity.com/paper/2502.01236/) (2025)
- [A Probabilistic Perspective on Unlearning and Alignment for Large Language Models](https://mlsanity.com/paper/2410.03523/) (2024)
- [Forecasting Rare Language Model Behaviors](https://mlsanity.com/paper/2502.16797/) (2025)
- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [Estimating Worst-Case Frontier Risks of Open-Weight LLMs](https://mlsanity.com/paper/2508.03153/) (2025)
- [Estimating the Probabilities of Rare Outputs in Language Models](https://mlsanity.com/paper/2410.13211/) (2024)
- [Sampling-aware Adversarial Attacks Against Large Language Models](https://mlsanity.com/paper/2507.04446/) (2025)
- [Rare Event Analysis of Large Language Models](https://mlsanity.com/paper/2602.06791/) (2026)
- [Predictive Querying for Autoregressive Neural Sequence Models](https://mlsanity.com/paper/2210.06464/) (2022)
- [Probabilistic Querying of Continuous-Time Event Sequences](https://mlsanity.com/paper/2211.08499/) (2022)

## Built on by (most cited first, 9 of 9)

- [Predicting LLM Safety Before Release by Simulating Deployment](https://mlsanity.com/paper/2607.07184/) (2026)
- [Automated Researchers Can Mitigate Well-characterized Alignment Failures](https://mlsanity.com/paper/2608.28945/) (2026)
- [Adaptive Multilevel Twisted Sequential Monte Carlo for Rare Events Estimation in Language Models](https://mlsanity.com/paper/2608.21736/) (2026)
- [Estimating Rare Events in Language Models with Proper Evaluation](https://mlsanity.com/paper/2607.18454/) (2026)
- [Sound Probabilistic Safety Bounds for Large Language Models](https://mlsanity.com/paper/2607.20286/) (2026)
- [Rare Event Estimation via Iterative Unalignment](https://mlsanity.com/paper/2609.24969/) (2026)
- [Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery](https://mlsanity.com/paper/2609.32547/) (2026)
- [Quantifying Behavioral Tails in Black-Box Language Models](https://mlsanity.com/paper/2609.33638/) (2026)
- [Estimating Great Expectations Under Autoregressive Language Models with Potentials](https://mlsanity.com/paper/2610.11399/) (2026)

## Abstract (arXiv)

Language models are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk