mlsanityData to 7 Oct 2026

WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs

arXiv 2406.18495 ↗First posted 26 Jun 2024cs.CLcited by 460

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

We introduce WildGuard -- an open, light-weight moderation tool for LLM safety that achieves three goals: (1) identifying malicious intent in user prompts, (2) detecting safety risks of model responses, and (3) determining model refusal rate. Together, WildGuard serves the increasing needs for automatic safety moderation and evaluation of LLM interactions, providing a one-stop tool with enhanced accuracy and broad coverage across 13 risk categories. While existing open moderation tools such as Llama-Guard2 score reasonably well in classifying straightforward model interactions, they lag far behind a prompted GPT-4, especially in identifying adversarial jailbreaks and in evaluating models' refusals, a key measure for evaluating safety behaviors in model responses. To address these challenges, we construct WildGuardMix, a large-scale and carefully balanced multi-task safety moderation dataset with 92K labeled examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired with various refusal and compliance responses. WildGuardMix is a combination of WildGuardTrain, the training data of WildGuard, and WildGuardTest, a high-quality human-annotated moderation test set with 5K labeled items covering broad risk scenarios. Through extensive evaluations on WildGuardTest and ten existing public benchmarks, we show that WildGuard establishes state-of-the-art performance in open-source safety moderation across all the three tasks compared to ten strong existing open-source moderation models (e.g., up to 26.4% improvement on refusal detection). Importantly, WildGuard matches and sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt harmfulness identification). WildGuard serves as a highly effective safety moderator in an LLM interface, reducing the success rate of jailbreak attacks from 79.8% to 2.4%.

Key facts. Cited by 460 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 28 of its 46 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

28 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming6 Apr

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  12. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
15 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  2. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  3. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  4. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  5. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  6. 2023An Overview of Catastrophic AI Riskscited by 214
  7. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 206
  8. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  9. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  10. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  11. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  12. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  13. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  14. 2020SOLID: A Large-Scale Semi-Supervised Dataset for Offensive Language Identificationcited by 53
  15. 2021Ruddit: Norms of Offensiveness for English Reddit Commentscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 165; 2026: 268.

460 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  4. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  5. 2025Qwen3Guard Technical Reportcited by 125
  6. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  7. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  8. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 97
  11. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  12. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
38 more
  1. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  2. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  3. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  4. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  5. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  6. 2025Safety in Large Reasoning Models: A Surveycited by 58
  7. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  8. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  9. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 50
  10. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  11. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  12. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  13. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  14. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  15. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  16. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  17. 2026Building Production-Ready Probes For Geminicited by 38
  18. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 37
  19. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  20. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  21. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirementscited by 35
  22. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  23. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  24. 2025Large-Scale Data Selection for Instruction Tuningcited by 34
  25. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  26. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  27. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  28. 2025MergeBench: A Benchmark for Merging Domain-Specialized LLMscited by 30
  29. 2025Representation Bending for Large Language Model Safetycited by 28
  30. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  31. 2025The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gainscited by 25
  32. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  33. 2025CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applicationscited by 24
  34. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  35. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  36. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  37. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 20
  38. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20

50 most cited of 460 All 460 (JSON)