# Estimating the Probabilities of Rare Outputs in Language Models

arXiv: https://arxiv.org/abs/2410.13211 · HTML: https://mlsanity.com/paper/2410.13211/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [FUDGE: Controlled Text Generation With Future Discriminators](https://mlsanity.com/paper/2104.05218/) (2021)
- [Are aligned neural networks adversarially aligned?](https://mlsanity.com/paper/2306.15447/) (2023)
- [Recent Advances in Adversarial Training for Adversarial Robustness](https://mlsanity.com/paper/2102.01356/) (2021)
- [Risks from Learned Optimization in Advanced Machine Learning Systems](https://mlsanity.com/paper/1906.01820/) (2019)
- [Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs](https://mlsanity.com/paper/2407.15549/) (2024)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals](https://mlsanity.com/paper/2210.01790/) (2022)
- [Adversarial Training for Large Neural Language Models](https://mlsanity.com/paper/2004.08994/) (2020)
- [Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo](https://mlsanity.com/paper/2404.17546/) (2024)
- [Defending Against Unforeseen Failure Modes with Latent Adversarial Training](https://mlsanity.com/paper/2403.05030/) (2024)
- [Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs](https://mlsanity.com/paper/2306.03081/) (2023)
- [A Statistical Approach to Assessing Neural Network Robustness](https://mlsanity.com/paper/1811.07209/) (2018)
- [Transfer of Adversarial Robustness Between Perturbation Types](https://mlsanity.com/paper/1905.01034/) (2019)
- [Analyzing Probabilistic Methods for Evaluating Agent Capabilities](https://mlsanity.com/paper/2409.16125/) (2024)

## Built on by (most cited first, 16 of 16)

- [Forecasting Rare Language Model Behaviors](https://mlsanity.com/paper/2502.16797/) (2025)
- [Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?](https://mlsanity.com/paper/2412.12480/) (2024)
- [Estimating Tail Risks in Language Model Output Distributions](https://mlsanity.com/paper/2604.22167/) (2026)
- [Rare Event Analysis of Large Language Models](https://mlsanity.com/paper/2602.06791/) (2026)
- [Predicting LLM Safety Before Release by Simulating Deployment](https://mlsanity.com/paper/2607.07184/) (2026)
- [Calibrated Predictive Lower Bounds on Time-to-Unsafe-Sampling in LLMs](https://mlsanity.com/paper/2506.13593/) (2025)
- [Towards Worst-Case Guarantees with Scale-Aware Interpretability](https://mlsanity.com/paper/2602.05184/) (2026)
- [Evaluating the Unseen Capabilities: How Many Theorems Do LLMs Know?](https://mlsanity.com/paper/2506.02058/) (2025)
- [SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddings](https://mlsanity.com/paper/2606.29623/) (2026)
- [Estimating Rare Events in Language Models with Proper Evaluation](https://mlsanity.com/paper/2607.18454/) (2026)
- [Training ML Models with Predictable Failures](https://mlsanity.com/paper/2605.15134/) (2026)
- [Sound Probabilistic Safety Bounds for Large Language Models](https://mlsanity.com/paper/2607.20286/) (2026)
- [BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditing](https://mlsanity.com/paper/2608.31105/) (2026)
- [Contract monitoring: governing AI via separation of powers](https://mlsanity.com/paper/2609.32061/) (2026)
- [Quantifying Behavioral Tails in Black-Box Language Models](https://mlsanity.com/paper/2609.33638/) (2026)
- [Priced Guidance: Can Language Models Generate Future Research Ideas?](https://mlsanity.com/paper/2610.04976/) (2026)

## Abstract (arXiv)

We consider the problem of low probability estimation: given a machine learning model and a formally-specified input distribution, how can we estimate the probability of a binary property of the model's output, even when that probability is too small to estimate by random sampling? This problem is motivated by the need to improve worst-case performance, which distribution shift can make much more likely. We study low probability estimation in the context of argmax sampling from small transformer language models. We compare two types of methods: importance sampling, which involves searching for inputs giving rise to the rare output, and activation extrapolation, which involves extrapolating a probability distribution fit to the model's logits. We find that importance sampling outperforms activation extrapolation, but both outperform naive sampling. Finally, we explain how minimizing the probability estimate of an undesirable behavior generalizes adversarial training, and argue that new methods for low probability estimation are needed to provide stronger guarantees about worst-case performance.