# I Can't Believe It's Not Robust: Catastrophic Collapse of Safety Classifiers under Embedding Drift

arXiv: https://arxiv.org/abs/2603.01297 · HTML: https://mlsanity.com/paper/2603.01297/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 1 Mar 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 5 of 5)

- [Towards evaluations-based safety cases for AI scheming](https://mlsanity.com/paper/2411.03336/) (2024)
- [Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)](https://mlsanity.com/paper/2501.19047/) (2025)
- [A benchmark for toxic comment classification on Civil Comments dataset](https://mlsanity.com/paper/2301.11125/) (2023)
- [Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations](https://mlsanity.com/paper/2511.05613/) (2025)
- [Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma](https://mlsanity.com/paper/2511.19504/) (2025)

## Built on by (most cited first, 3 of 3)

- [Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers](https://mlsanity.com/paper/2606.11949/) (2026)
- [Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models](https://mlsanity.com/paper/2606.08571/) (2026)
- [Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks](https://mlsanity.com/paper/2608.09624/) (2026)

## Abstract (arXiv)

Instruction tuned reasoning models are increasingly deployed with safety classifiers trained on frozen embeddings, assuming representation stability across model updates. We systematically investigate this assumption and find it fails: normalized perturbations of magnitude σ=0.02 (corresponding to ≈ 1^circ angular drift on the embedding sphere) reduce classifier performance from 85% to 50% ROC-AUC. Critically, mean confidence only drops 14%, producing dangerous silent failures where 72% of misclassifications occur with high confidence, defeating standard monitoring. We further show that instruction-tuned models exhibit 20% worse class separability than base models, making aligned systems paradoxically harder to safeguard. Our findings expose a fundamental fragility in production AI safety architectures and challenge the assumption that safety mechanisms transfer across model versions.