mlsanityData to 7 Oct 2026

AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Experts

arXiv 2404.05993 ↗First posted 9 Apr 2024cs.LG cs.CL cs.CYcited by 126

Not on the map. Why?

Cited by 70 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

As Large Language Models (LLMs) and generative AI become more widespread, the content safety risks associated with their use also increase. We find a notable deficiency in high-quality content safety datasets and benchmarks that comprehensively cover a wide range of critical safety areas. To address this, we define a broad content safety risk taxonomy, comprising 13 critical risk and 9 sparse risk categories. Additionally, we curate AEGISSAFETYDATASET, a new dataset of approximately 26, 000 human-LLM interaction instances, complete with human annotations adhering to the taxonomy. We plan to release this dataset to the community to further research and to help benchmark LLM models for safety. To demonstrate the effectiveness of the dataset, we instruction-tune multiple LLM-based safety models. We show that our models (named AEGISSAFETYEXPERTS), not only surpass or perform competitively with the state-of-the-art LLM-based safety models and general purpose LLMs, but also exhibit robustness across multiple jail-break attack categories. We also show how using AEGISSAFETYDATASET during the LLM alignment phase does not negatively impact the performance of the aligned models on MT Bench scores. Furthermore, we propose AEGIS, a novel application of a no-regret online adaptation framework with strong theoretical guarantees, to perform content moderation with an ensemble of LLM content safety experts in deployment

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 9 Apr 2024.

Builds on

22 of 32 references.

References · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2023Mistral 7Bcited by 3,065
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  12. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
10 more
  1. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  2. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  4. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  5. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  6. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  7. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  8. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63
  9. 2023BiasX: "Thinking Slow" in Toxic Content Moderation with Explanations of Implied Social Biasescited by 7
  10. 2023Cappy: Outperforming and Boosting Large Multi-Task LMs with a Small Scorercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 54; 2026: 51.

126 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  3. 2024Nemotron-4 340B Technical Reportcited by 143
  4. 2025Qwen3Guard Technical Reportcited by 125
  5. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  6. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 118
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  8. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  9. 2025Safety in Large Reasoning Models: A Surveycited by 58
  10. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  11. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  12. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
38 more
  1. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 41
  2. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 38
  3. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrailscited by 29
  4. 2024A Survey of Useful LLM Evaluationcited by 28
  5. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  6. 2024Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpuscited by 24
  7. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  8. 2025CultureGuard: Towards Culturally-Aware Dataset and Guard Model for Multilingual Safety Applicationscited by 24
  9. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 23
  10. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  11. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  12. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 20
  13. 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
  14. 2025Enhancing LLM Safety Through a Theoretical Minimax Game Lenscited by 18
  15. 2024HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Modelscited by 17
  16. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 17
  17. 2024MEOW: MEMOry Supervised LLM Unlearning Via Inverted Factscited by 16
  18. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  19. 2026YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Modelscited by 14
  20. 2024Benchmarking LLM Guardrails in Handling Multilingual Toxicitycited by 13
  21. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  22. 2025Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderationcited by 12
  23. 2024STAND-Guard: A Small Task-Adaptive Content Moderation Modelcited by 11
  24. 2024SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teamingcited by 10
  25. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  26. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  27. 2026LLM Safety From Within: Detecting Harmful Content with Internal Representationscited by 10
  28. 2024The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Modelscited by 9
  29. 2025Detecting and Filtering Unsafe Training Data via Data Attribution with Denoised Representationcited by 9
  30. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  31. 2025ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Modelscited by 9
  32. 2024Lightweight Safety Guardrails Using Fine-tuned BERT Embeddingscited by 8
  33. 2024SLM as Guardian: Pioneering AI Safety with Small Language Modelscited by 7
  34. 2025RealHarm: A Collection of Real-World Language Model Application Failurescited by 7
  35. 2024SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMscited by 6
  36. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  37. 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
  38. 2024Lightweight Safety Classification Using Pruned Language Modelscited by 5

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model Safety: A Holistic Survey
  2. 2024AI Safety in Generative AI Large Language Models: A Survey
  3. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  4. 2025LLM-Safety Evaluations Lack Robustness
  5. 2025Probing AI Safety with Source Code
  6. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types
  7. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
  8. 2025aiXamine: Simplified LLM Safety and Security