# Regime-Conditional Verification: Correctness Estimation for Adapting and Monitoring Safety Classifiers

arXiv: https://arxiv.org/abs/2608.14089 · HTML: https://mlsanity.com/paper/2608.14089/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 22 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Builds on (most cited first, 19 of 19)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Class-Conditional Conformal Prediction with Many Classes](https://mlsanity.com/paper/2306.09335/) (2023)
- [Safety Cases: How to Justify the Safety of Advanced AI Systems](https://mlsanity.com/paper/2403.10462/) (2024)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalize](https://mlsanity.com/paper/2509.03888/) (2025)
- [Unsupervised Concept Drift Detection from Deep Learning Representations in Real-time](https://mlsanity.com/paper/2406.17813/) (2024)
- [WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales](https://mlsanity.com/paper/2505.04608/) (2025)
- [Confidence-based Estimators for Predictive Performance in Model Monitoring](https://mlsanity.com/paper/2407.08649/) (2024)
- [JailbreaksOverTime: Detecting Jailbreak Attacks Under Distribution Shift](https://mlsanity.com/paper/2504.19440/) (2025)
- [Fair Wrapping for Black-box Predictions](https://mlsanity.com/paper/2201.12947/) (2022)
- [DriftGuard: Safety-Aware Multi-Monitor Detection and Selective Adaptation for Evolving Toxicity Moderation](https://mlsanity.com/paper/2606.28725/) (2026)
- [RepV: Safety-Separable Latent Spaces for Scalable Neurosymbolic Plan Verification](https://mlsanity.com/paper/2510.26935/) (2025)

## Abstract (arXiv)

Safety classifiers deployed with large language models often fail for two reasons: their decisions reflect the policy learned during training rather than the deployer's desired policy, and their performance degrades as deployment traffic evolves. We present Regime-Conditional Verification (RCV), a lightweight wrapper that adapts an off-the-shelf safety classifier without retraining it. RCV estimates, from the classifier's internal representations, the probability that each prediction disagrees with the deployer's policy, and selectively corrects predictions likely to be wrong. The same correctness estimates also provide a label-free signal for detecting distribution shift, enabling a maintenance loop that updates the correctness estimation layer and resorts to classifier fine-tuning only when repair fails within a label budget. Across three off-the-shelf safety classifiers and two benchmark datasets, RCV improves adherence to the deployer's policy in every classifier-dataset combination, catching up to 0.81 of previously missed unsafe content without modifying the underlying classifier. In a deployment study with ten attack campaigns, each a harm category held out of RCV's training, RCV detects every campaign in a dedicated injection panel; in the maintenance census most drift episodes are repaired without updating the classifier, and the fine-tune is reserved for the residual episodes.