Safety Monitors Mostly Catch What the Model Already Refuses
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
Safety monitors are evaluated by recall on harmful prompts, regardless of whether the target model would answer them. Yet a monitor matters most on the prompts the model does answer. We measure recall on exactly those prompts, defined by sampling the target model and judging its responses. Across four text guards, two activation probes, and Latent Guard, recall at a 1% false positive rate falls sharply on this subset: at a common threshold, every monitor catches the requests the model refuses 1.1 to 6.4 times as often as the requests it answers. Standard metrics hide this; AUROC stays above 0.85 for most monitors. Rewriting each request to be less explicit, with intent held fixed and verified, raises compliance 28-fold and lowers every monitor's flag rate. Of the requests newly answered after rewriting, 44 to 93% slip past the monitor, depending on which is used, and most of their completions are graded harmful. We trace the gap to explicitness itself. As wording softens with intent fixed, the target model's harm and refusal readings fall and it answers; the guards' harm readings fall too, and steering a guard along explicitness alone flips its verdict. Model and monitors miss the same prompts, and stacking monitors does not recover them. Fine-tuning a guard on the rewrites at every level of explicitness, on both sides of the label, raises recall on answered requests from .24 to .89 while transferring to unseen benchmarks; hard negatives, the natural alternative, teach the guard to discount indirect phrasing instead.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 33 of its 46 references that are in the map. First posted on arXiv 5 Sep 2026.
Builds on
33 of 46 references.
Closest ancestors · 12
Same direction, newest first
- 2026Reflect-Guard: Enhancing LLM Safeguards against Adversarial Prompts via Logical Self-Reflection24 May
- 2025Guarding the Meaning: Self-Supervised Training for Semantic Robustness in Guard Models6 Nov
- 2025Qwen3Guard Technical Report16 Oct
- 2025LLMs Encode Harmfulness and Refusal Separately16 Jul
- 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence24 Feb
- 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions13 Feb
- 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails15 Jan
- 2024On Calibration of LLM-based Guard Models for Reliable Content Moderation14 Oct
- 2024Does Refusal Training in LLMs Generalize to the Past Tense?16 Jul
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun
- 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
- 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
Other references · 21
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2025Qwen3 Technical Reportcited by 8,820
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
- 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
- 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
- 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 439
- 2026Gemma 4 Technical Reportcited by 259
9 more
- 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
- 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
- 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
- 2025Detecting High-Stakes Interactions with Activation Probescited by 53
- 2026Building Production-Ready Probes For Geminicited by 47
- 2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 41
- 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 19
- 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
- 2026Latent Undertow: How Ordinary Typos Break Probescited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Behavioral Safety Evaluation Fails: A Representation-Level Perspective
- 2026Monitorability Disposition in Large Reasoning Models
- 2026What the Guard Misses, the Robot Executes: Implied Harm in VLA Instructions
- 2026Separating Prompt Harmfulness from Interaction Safety via Partial Generation
- 2026Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity
- 2026Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries
- 2026StepShield: When, Not Whether to Intervene on Rogue Agents
- 2026Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Staleness