# Rare Event Analysis of Large Language Models

arXiv: https://arxiv.org/abs/2602.06791 · HTML: https://mlsanity.com/paper/2602.06791/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 62 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Efficient Training of Language Models to Fill in the Middle](https://mlsanity.com/paper/2207.14255/) (2022)
- [A Thorough Examination of Decoding Methods in the Era of LLMs](https://mlsanity.com/paper/2402.06925/) (2024)
- [Reasoning with Sampling: Your Base Model is Smarter Than You Think](https://mlsanity.com/paper/2510.14901/) (2025)
- [Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs](https://mlsanity.com/paper/2306.03081/) (2023)
- [Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo](https://mlsanity.com/paper/2504.13139/) (2025)
- [Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening](https://mlsanity.com/paper/2601.21590/) (2026)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [A reinforcement learning approach to rare trajectory sampling](https://mlsanity.com/paper/2005.12890/) (2020)
- [Sample, Don't Search: Rethinking Test-Time Alignment for Language Models](https://mlsanity.com/paper/2504.03790/) (2025)
- [Forecasting Rare Language Model Behaviors](https://mlsanity.com/paper/2502.16797/) (2025)
- [QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation](https://mlsanity.com/paper/2406.00049/) (2024)
- [Estimating the Probabilities of Rare Outputs in Language Models](https://mlsanity.com/paper/2410.13211/) (2024)
- [Reinforcement learning of rare diffusive dynamics](https://mlsanity.com/paper/2105.04321/) (2021)
- [Combining Reinforcement Learning and Tensor Networks, with an Application to Dynamical Large Deviations](https://mlsanity.com/paper/2209.14089/) (2022)
- [Simple Confidence Intervals for MCMC Without CLTs](https://mlsanity.com/paper/1812.00126/) (2018)
- [Training neural network ensembles via trajectory sampling](https://mlsanity.com/paper/2209.11116/) (2022)
- [Bayesian Multistate Bennett Acceptance Ratio Methods](https://mlsanity.com/paper/2310.20699/) (2023)

## Built on by (most cited first, 8 of 8)

- [Estimating Tail Risks in Language Model Output Distributions](https://mlsanity.com/paper/2604.22167/) (2026)
- [Adaptive Multilevel Twisted Sequential Monte Carlo for Rare Events Estimation in Language Models](https://mlsanity.com/paper/2608.21736/) (2026)
- [Estimating Rare Events in Language Models with Proper Evaluation](https://mlsanity.com/paper/2607.18454/) (2026)
- [Training ML Models with Predictable Failures](https://mlsanity.com/paper/2605.15134/) (2026)
- [Rare Event Estimation via Iterative Unalignment](https://mlsanity.com/paper/2609.24969/) (2026)
- [Prioritizing Repeated LLM Evaluation for Hidden Failure Discovery](https://mlsanity.com/paper/2609.32547/) (2026)
- [Quantifying Behavioral Tails in Black-Box Language Models](https://mlsanity.com/paper/2609.33638/) (2026)
- [Estimating Great Expectations Under Autoregressive Language Models with Potentials](https://mlsanity.com/paper/2610.11399/) (2026)

## Abstract (arXiv)

Being probabilistic models, during inference large language models (LLMs) display rare events: behaviour that is far from typical but highly significant. By definition all rare events are hard to see, but the enormous scale of LLM usage means that events completely unobserved during development are likely to become prominent in deployment. Here we present an end-to-end framework for the systematic analysis of rare events in LLMs. We provide a practical implementation spanning theory, efficient generation strategies, probability estimation and error analysis, which we illustrate with concrete examples. We outline extensions and applications to other models and contexts, highlighting the generality of the concepts and techniques presented here.