mlsanityData to 7 Oct 2026

Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails

arXiv 2501.09004 ↗First posted 15 Jan 2025cs.CLcited by 122

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

As Large Language Models (LLMs) and generative AI become increasingly widespread, concerns about content safety have grown in parallel. Currently, there is a clear lack of high-quality, human-annotated datasets that address the full spectrum of LLM-related safety risks and are usable for commercial applications. To bridge this gap, we propose a comprehensive and adaptable taxonomy for categorizing safety risks, structured into 12 top-level hazard categories with an extension to 9 fine-grained subcategories. This taxonomy is designed to meet the diverse requirements of downstream users, offering more granular and flexible tools for managing various risk types. Using a hybrid data generation pipeline that combines human annotations with a multi-LLM "jury" system to assess the safety of responses, we obtain Aegis 2.0, a carefully curated collection of 34,248 samples of human-LLM interactions, annotated according to our proposed taxonomy. To validate its effectiveness, we demonstrate that several lightweight models, trained using parameter-efficient techniques on Aegis 2.0, achieve performance competitive with leading safety models fully fine-tuned on much larger, non-commercial datasets. In addition, we introduce a novel training blend that combines safety with topic following data.This approach enhances the adaptability of guard models, enabling them to generalize to new risk categories defined during inference. We plan to open-source Aegis 2.0 data and models to the research community to aid in the safety guardrailing of LLMs.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 25 of its 33 references that are in the map. First posted on arXiv 15 Jan 2025.

Builds on

25 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs26 Jun

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  9. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  10. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
12 more
  1. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  2. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  4. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 184
  5. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  6. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  7. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 126
  8. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  9. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  10. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applicationscited by 53
  11. 2024CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialoguescited by 15
  12. 2023BiasX: "Thinking Slow" in Toxic Content Moderation with Explanations of Implied Social Biasescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 47; 2026: 75.

122 papers.

Sort:

By area:

  1. 2025Qwen3Guard Technical Reportcited by 125
  2. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  3. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  4. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  5. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  6. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  7. 2025Safety in Large Reasoning Models: A Surveycited by 58
  8. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  9. 2026Meta Context Engineering via Agentic Skill Evolutioncited by 42
  10. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  11. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  12. 2025CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applicationscited by 24
38 more
  1. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  2. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  3. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 16
  4. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 15
  5. 2025SAFEx: Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identificationcited by 14
  6. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representationscited by 10
  7. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  8. 2025ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Modelscited by 9
  9. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  10. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  11. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  12. 2025LionGuard 2: Building Lightweight, Data-Efficient & Localised Multilingual Content Moderatorscited by 7
  13. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Trainingcited by 7
  14. 2025VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safetycited by 7
  15. 2026FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderationcited by 7
  16. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  17. 2025Kelp: A Streaming Safeguard for Large Models via Latent Dynamics-Guided Risk Detectioncited by 6
  18. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 6
  19. 2025ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMscited by 5
  20. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 5
  21. 2025LongRM: Revealing and Unlocking the Context Boundary of Reward Modelingcited by 5
  22. 2026UbuntuGuard: A Culturally-Grounded Policy Benchmark for Equitable AI Safety in African Languagescited by 5
  23. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systemscited by 5
  24. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 4
  25. 2026GLiGuard: Schema-Conditioned Classification for LLM Safeguardcited by 4
  26. 2026LPG: Balancing Efficiency and Policy Reasoning in Latent Policy Guardrailscited by 4
  27. 2026Monitoring the Internal Monologue: Probe Trajectories Reveal Reasoning Dynamicscited by 4
  28. 2025SGuard-v1: Safety Guardrail for Large Language Modelscited by 3
  29. 2026Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignmentcited by 3
  30. 2026Robust Policy Optimization to Prevent Catastrophic Forgettingcited by 3
  31. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  32. 2026Token-level Data Selection for Safe LLM Fine-tuningcited by 3
  33. 2026NExT-Guard: Training-Free Streaming Safeguard without Token-Level Labelscited by 3
  34. 2025The Structural Safety Generalization Problemcited by 2
  35. 2025PAM: Training Policy-Aligned Moderation Filters at Scalecited by 2
  36. 2025Attention-Aware GNN-based Input Defense against Multi-Turn LLM Jailbreakcited by 2
  37. 2025Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMscited by 2
  38. 2025AdaptiveGuard: Towards Adaptive Runtime Safety for LLM-Powered Softwarecited by 2

50 most cited of 122 All 122 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Current state of LLM Risks and AI Guardrails
  2. 2025GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
  3. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets
  4. 2026CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment
  5. 2025LLM-Safety Evaluations Lack Robustness
  6. 2024Concept-Guided LLM Agents for Human-AI Safety Codesign
  7. 2026Beyond Unsafe Detection: Counterfactually Anchored Evidence Attribution for Multi-Turn LLM Safety Failures
  8. 2026Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools