Online Safety Monitoring for LLMs
Not on the map. Why?
Abstract (arXiv)
Despite alignment training, LLMs remain prone to generating unsafe outputs at deployment time. Monitoring outputs online and raising an alarm when safety can no longer be assumed is therefore critical. We study a simple real-time monitor that turns a verifier signal from an external model into an alarm decision by thresholding, with the threshold calibrated via risk control. In experiments on mathematical reasoning and red teaming datasets, we show that this simple design is competitive with more advanced monitors based on sequential hypothesis testing.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 57 of its 66 references that are in the map. First posted on arXiv 2 Jul 2026.
Builds on
57 of 66 references.
References · 57
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2023Let's Verify Step by Stepcited by 3,598
- 2023Mistral 7Bcited by 3,065
- 2023A Survey of Large Language Modelscited by 2,537
- 2022Survey of Hallucination in Natural Language Generationcited by 2,056
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2021Ethical and social risks of harm from Language Modelscited by 957
- 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
38 more
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
- 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
- 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
- 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
- 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
- 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
- 2022Conformal Risk Controlcited by 290
- 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
- 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
- 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
- 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetycited by 222
- 2018Time-uniform, nonparametric, nonasymptotic confidence sequencescited by 220
- 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
- 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
- 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
- 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 172
- 2023Conformal Language Modelingcited by 163
- 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
- 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
- 2022Game-theoretic statistics and safe anytime-valid inferencecited by 145
- 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
- 2024Language Models with Conformal Factuality Guaranteescited by 128
- 2024What Large Language Models Know and What People Think They Knowcited by 109
- 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
- 2024Hypothesis testing with e-valuescited by 70
- 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
- 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
- 2021Tracking the risk of a deployed model and detecting harmful distribution shiftscited by 38
- 2022E-detectors: a nonparametric framework for sequential change detectioncited by 27
- 2024Fast yet Safe: Early-Exiting with Risk Controlcited by 27
- 2024Protected Test-Time Adaptation via Online Entropy Matching: A Betting Approachcited by 22
- 2022Efficiently Controlling Multiple Risks with Pareto Testingcited by 21
- 2025Thought calibration: Efficient and confident test-time scalingcited by 20
- 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
- 2025E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testingcited by 14
- 2025WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingalescited by 13
- 2025Probabilistic Soundness Guarantees in LLM Reasoning Chainscited by 13
50 most cited of 57 All 57 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Why Formal Monitors Fail: Attack Distribution Entropy as a Coverage Bound for LTL-Based LLM Agent Safety
- 2026HESP: Separating What to Probe from When to Stop in Local LLM Alert-Triage Agents
- 2026PreAct-Bench: Benchmarking Predictive Monitoring in LLMs
- 2026Real-Time Detection and Repair of LLM Agent Failures
- 2026NEXUS: Structured Runtime Safety for Tool-Using LLM Agents
- 2025Saffron-1: Safety Inference Scaling
- 2025Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance
- 2026MonitorVLM-v2: A Deployed Vision-Language Framework for Real-Time Safety Violation Detection