Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming
On the map: LLM safety and security → Safety alignment, refusal and guardrails
Abstract (arXiv)
In many practical LLM deployments, a single guardrail is used for both prompt and response moderation. Prompt moderation operates on fully observed text, whereas streaming response moderation requires safety decisions to be made over partial generations. Existing text-based streaming guardrails commonly frame this output-side problem as boundary detection, training models to identify the earliest prefix at which a response has already become unsafe. In this work, we introduce StreamGuard, a unified model-agnostic streaming guardrail that instead formulates moderation as a forecasting problem: given a partial prefix, the model predicts the expected harmfulness of likely future continuations. We supervise this prediction using Monte Carlo rollouts, which enables early intervention without requiring exact token-level boundary annotations. Across standard safety benchmarks, StreamGuard performs strongly both for input moderation and for streaming output moderation. At the 8B scale, StreamGuard improves aggregated input-moderation F1 from 86.7 to 88.2 and aggregated streaming output-moderation F1 from 80.4 to 81.9 relative to Qwen3Guard-Stream-8B-strict. On the QWENGUARDTEST response_loc streaming benchmark, StreamGuard reaches 97.5 F1, 95.1 recall, and 92.6% on-time intervention, compared to 95.9 F1, 92.1 recall, and 89.9% for Qwen3Guard-Stream-8B-stric, while reducing the miss rate from 7.9% to 4.9%. We further show that forecasting-based supervision transfers effectively across tokenizers and model families: with transferred targets, Gemma3-StreamGuard-1B reaches 81.3 response-moderation F1, 98.2 streaming F1, and a 3.5% miss rate. These results show that strong end-to-end streaming moderation can be obtained without exact boundary labels, and that forecasting future risk is an effective supervision strategy for low-latency safety intervention.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 17 of its 24 references that are in the map. First posted on arXiv 5 Apr 2026.
Builds on
17 of 24 references.
Closest ancestors · 4
Same direction, newest first
- 2025Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detection9 Oct
- 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policies2 Sep
- 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails15 Jan
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun
Other references · 13
- 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
- 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
- 2023Are aligned neural networks adversarially aligned?cited by 315
- 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
- 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
- 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
- 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
- 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 76
- 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 59
1 more
- 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 21
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026AERIC: Anticipatory Hidden-State Monitoring for Implicit Harmful Dialoguecited by 0
- 2026FreoStream:Enhancing Stream Guardrails via Future-Aware Reasoning and Safety-Aligned Optimizationcited by 0
- 2026Withholding the Completing Chunk: Exact Release-Boundary Equivalence for Production Streaming Guardrailscited by 0
- 2026Withholding the Completing Chunk: Exact Release-Boundary Equivalence for Production Streaming Guardrails10 Aug 2026
- 2026FreoStream:Enhancing Stream Guardrails via Future-Aware Reasoning and Safety-Aligned Optimization11 Jun 2026
- 2026AERIC: Anticipatory Hidden-State Monitoring for Implicit Harmful Dialogue13 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring
- 2026ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributions
- 2026FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderation
- 2026Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs
- 2026NExT-Guard: Training-Free Streaming Safeguard without Token-Level Labels
- 2026SentGuard: Sentence-Level Streaming Guardrails for Large Language Models
- 2025Protect: Towards Robust Guardrailing Stack for Trustworthy Enterprise LLM Systems
- 2024On Calibration of LLM-based Guard Models for Reliable Content Moderation