mlsanityData to 14 Oct 2026

Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

arXiv 2312.06674 ↗First posted 7 Dec 2023cs.CL cs.AIcited by 1,207

Not on the map. Why?

Cited by 618 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk taxonomy, a valuable tool for categorizing a specific set of safety risks found in LLM prompts (i.e., prompt classification). This taxonomy is also instrumental in classifying the responses generated by LLMs to these prompts, a process we refer to as response classification. For the purpose of both prompt and response classification, we have meticulously gathered a dataset of high quality. Llama Guard, a Llama2-7b model that is instruction-tuned on our collected dataset, albeit low in volume, demonstrates strong performance on existing benchmarks such as the OpenAI Moderation Evaluation dataset and ToxicChat, where its performance matches or exceeds that of currently available content moderation tools. Llama Guard functions as a language model, carrying out multi-class classification and generating binary decision scores. Furthermore, the instruction fine-tuning of Llama Guard allows for the customization of tasks and the adaptation of output formats. This feature enhances the model's capabilities, such as enabling the adjustment of taxonomy categories to align with specific use cases, and facilitating zero-shot or few-shot prompting with diverse taxonomies at the input. We are making Llama Guard model weights available and we encourage researchers to further develop and adapt them to meet the evolving needs of the community for AI safety.

Key facts. Cited by 1,207 papers in the map (data through 14 Oct 2026). Builds on 17 of its 26 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

17 of 26 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2023PaLM 2 Technical Reportcited by 1,162
  8. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  9. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  10. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  11. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 207
  12. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
5 more
  1. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  2. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  3. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 78
  4. 2021Ruddit: Norms of Offensiveness for English Reddit Commentscited by 27
  5. 2020Preserving Integrity in Online Social Networkscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 258; 2025: 384; 2026: 564.

1,207 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  2. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  7. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  8. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  9. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  10. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  11. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  12. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
38 more
  1. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  2. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  3. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  4. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  5. 2025Qwen3Guard Technical Reportcited by 155
  6. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 151
  7. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 145
  8. 2024Nemotron-4 340B Technical Reportcited by 144
  9. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  10. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  11. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  12. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  13. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 124
  14. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  15. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  16. 2024Rule Based Rewards for Language Model Safetycited by 108
  17. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  18. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 101
  19. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 100
  20. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  21. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
  22. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  23. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 96
  24. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  25. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  26. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 91
  27. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  28. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  29. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 87
  30. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  31. 2024SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Modelcited by 85
  32. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  33. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  34. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80
  35. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  36. 2024Safeguarding Large Language Models: A Surveycited by 78
  37. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  38. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 77

50 most cited of 1,207 All 1,207 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards
  2. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
  3. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
  4. 2025DeepKnown-Guard: A Proprietary Model-Based Safety Response Framework for AI Agents
  5. 2026AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety
  6. 2026Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation
  7. 2025Policy-Grounded Safety Evaluation of 20 Large Language Models
  8. 2026Detection of adversarial intent in Human-AI teams using LLMs