# Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs

arXiv: https://arxiv.org/abs/2605.21602 · HTML: https://mlsanity.com/paper/2605.21602/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 20 May 2026.

## Builds on (most cited first, 22 of 22)

- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking](https://mlsanity.com/paper/2312.09244/) (2023)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shifts](https://mlsanity.com/paper/2410.10700/) (2024)
- [Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection](https://mlsanity.com/paper/2101.03778/) (2021)
- [On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback](https://mlsanity.com/paper/2411.02306/) (2024)
- [Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection](https://mlsanity.com/paper/2406.00806/) (2024)
- [Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks](https://mlsanity.com/paper/2210.07907/) (2022)
- [Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?](https://mlsanity.com/paper/2412.03235/) (2024)
- [LLM Jailbreak Detection for (Almost) Free!](https://mlsanity.com/paper/2509.14558/) (2025)
- [JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Model](https://mlsanity.com/paper/2504.03770/) (2025)
- [Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks](https://mlsanity.com/paper/2511.22047/) (2025)
- [Learning to Detect Unseen Jailbreak Attacks in Large Vision-Language Models](https://mlsanity.com/paper/2508.09201/) (2025)

## Built on by (most cited first, 4 of 4)

- [Toward Mechanistic Interpretability of an AI Foundation Model Fine-Tuned for Atmospheric Chemistry](https://mlsanity.com/paper/2607.20778/) (2026)
- [A Safe Prototype Is Not a Safety Direction: Reference Dependence and Prompt Confounds in Response-Safety Embeddings](https://mlsanity.com/paper/2610.01801/) (2026)
- [Strategic Governance of AI Models in Earth Science](https://mlsanity.com/paper/2610.10560/) (2026)
- [Genesis Steering: Better Activation Steering Through Training Data](https://mlsanity.com/paper/2610.19114/) (2026)

## Abstract (arXiv)

Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these OOD alignment failures by introducing a benchmark called Misalignment Out Of Distribution (MOOD). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model with Mahalanobis distance and perplexity-based OOD detectors can improve recall from 39% to 45%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem. We release the code and data for our experiments publicly, and you can find the relevant links here: https://github.com/Dylan102938/mood-bench.