mlsanityData to 14 Oct 2026

From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring

arXiv 2506.09996 ↗First posted 11 Jun 2025cs.CL cs.CYcited by 22

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Though safety alignment has been applied to most large language models (LLMs), LLM service providers generally deploy a subsequent moderation as the external safety guardrail in real-world products. Existing moderators mainly practice a conventional full detection, which determines the harmfulness based on the complete LLM output, causing high service latency. Recent works pay more attention to partial detection where moderators oversee the generation midway and early stop the output if harmfulness is detected, but they directly apply moderators trained with the full detection paradigm to incomplete outputs, introducing a training-inference gap that lowers the performance. In this paper, we explore how to form a data-and-model solution that natively supports partial detection. For the data, we construct FineHarm, a dataset consisting of 29K prompt-response pairs with fine-grained annotations to provide reasonable supervision for token-level training. Then, we propose the streaming content monitor, which is trained with dual supervision of response- and token-level labels and can follow the output stream of LLM to make a timely judgment of harmfulness. Experiments show that SCM gains 0.95+ in macro F1 score that is comparable to full detection, by only seeing the first 18% of tokens in responses on average. Moreover, the SCM can serve as a pseudo-harmfulness annotator for improving safety alignment and lead to a higher harmlessness score than DPO.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 62 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

43 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing4 Feb
  2. 2024Imposter.AI: Adversarial Attacks with Hidden Intentions towards Aligned Large Language Models22 Jul
  3. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  4. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  5. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  6. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  8. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  11. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  12. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
23 more
  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  2. 2023Detecting Language Model Attacks with Perplexitycited by 431
  3. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  4. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 407
  5. 2024Alignment faking in large language modelscited by 360
  6. 2023Certifying LLM Safety against Adversarial Promptingcited by 266
  7. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  8. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 223
  9. 2020HateBERT: Retraining BERT for Abusive Language Detection in Englishcited by 207
  10. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 183
  11. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  12. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  13. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  14. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  15. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  16. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  17. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  18. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  19. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characterscited by 42
  20. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 34
  21. 2024Is the System Message Really Important to Jailbreaks in Large Language Models?cited by 20
  22. 2024T-REG: Preference Optimization with Token-Level Reward Regularizationcited by 13
  23. 2020Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detectioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 15.

22 papers.

Sort:

By area:

  1. 2025Qwen3Guard Technical Reportcited by 155
  2. 2025Beyond Linear Probes: Dynamic Safety Monitoring for Language Modelscited by 16
  3. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  4. 2025Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detectioncited by 9
  5. 2026NExT-Guard: Training-Free Streaming Safeguard without Token-Level Labelscited by 4
  6. 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depthcited by 3
  7. 2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
  8. 2026SentGuard: Sentence-Level Streaming Guardrails for Large Language Modelscited by 2
  9. 2025Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attackscited by 1
  10. 2025Sanitize Your Responses: Mitigating Privacy Leakage in Large Language Modelscited by 1
  11. 2026ROM: Real-time Overthinking Mitigation via Streaming Detection and Interventioncited by 1
  12. 2026TwinGate: Stateful Defense against Decompositional Jailbreaks in Untraceable Traffic via Asymmetric Contrastive Learningcited by 1
10 more
  1. 2026Stop Testing Attacks, Start Diagnosing Defenses: The Four-Checkpoint Framework Reveals Where LLM Safety Breakscited by 0
  2. 2026DART: Mitigating Harm Drift in Difference-Aware LLMs via Distill-Audit-Repair Trainingcited by 0
  3. 2026AERIC: Anticipatory Hidden-State Monitoring for Implicit Harmful Dialoguecited by 0
  4. 2026FreoStream:Enhancing Stream Guardrails via Future-Aware Reasoning and Safety-Aligned Optimizationcited by 0
  5. 2026Online Safety Monitoring for LLMscited by 0
  6. 2026Operational Evidence Gaps for LLMs in Fraud Detection and Trust-and-Safety Workflowscited by 0
  7. 2026ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributionscited by 0
  8. 2026SingProbe Technical Reportcited by 0
  9. 2026ForeSight: Enhancing Risk Monitoring via Early Safety Signal Distillationcited by 0
  10. 2026Separating Prompt Harmfulness from Interaction Safety via Partial Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  2. 2026Predict, Don't React: Value-Based Safety Forecasting for LLM Streaming
  3. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectors
  4. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representations
  5. 2026Improving LLM Reliability through Hybrid Abstention and Adaptive Detection
  6. 2026Stop Early, Spend Less: Hidden-State Probes as a Practical Recipe for Streaming Moderation of LLM Outputs
  7. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  8. 2025LLM-Safety Evaluations Lack Robustness