mlsanityData to 14 Oct 2026

Introducing v0.5 of the AI Safety Benchmark from MLCommons

arXiv 2404.12241 ↗First posted 18 Apr 2024cs.CL cs.AIcited by 62

Not on the map. Why?

Abstract (arXiv)

This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safety risks of AI systems that use chat-tuned language models. We introduce a principled approach to specifying and constructing the benchmark, which for v0.5 covers only a single use case (an adult chatting to a general-purpose assistant in English), and a limited set of personas (i.e., typical users, malicious users, and vulnerable users). We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark. We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024. The v1.0 benchmark will provide meaningful insights into the safety of AI systems. However, the v0.5 benchmark should not be used to assess the safety of AI systems. We have sought to fully document the limitations, flaws, and challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes (1) a principled approach to specifying and constructing the benchmark, which comprises use cases, types of systems under test (SUTs), language and context, personas, tests, and test items; (2) a taxonomy of 13 hazard categories with definitions and subcategories; (3) tests for seven of the hazard categories, each comprising a unique set of test items, i.e., prompts. There are 43,090 test items in total, which we created with templates; (4) a grading system for AI systems against the benchmark; (5) an openly available platform, and downloadable tool, called ModelBench that can be used to evaluate the safety of AI systems on the benchmark; (6) an example evaluation report which benchmarks the performance of over a dozen openly available chat-tuned language models; (7) a test specification for the benchmark.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Builds on 73 of its 180 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

73 of 180 references.

References · 73

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
38 more
  1. 2023PaLM 2 Technical Reportcited by 1,162
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  7. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  11. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  12. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  13. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  14. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  15. 2024A StrongREJECT for Empty Jailbreakscited by 385
  16. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  17. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  18. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  19. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  20. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  21. 2023An Overview of Catastrophic AI Riskscited by 222
  22. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  23. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 207
  24. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  25. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 199
  26. 2023Model evaluation for extreme riskscited by 177
  27. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  28. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  29. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 165
  30. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 165
  31. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  32. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  33. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  34. 2023Evaluating and Mitigating Discrimination in Language Model Decisionscited by 124
  35. 2019MLPerf Training Benchmarkcited by 115
  36. 2024Cognitive Bias in Decision-Making with LLMscited by 106
  37. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 95
  38. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 23; 2026: 18.

62 papers.

Sort:

By area:

  1. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  2. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 133
  3. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  4. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  5. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  6. 2025Establishing Best Practices for Building Rigorous Agentic Benchmarkscited by 102
  7. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  8. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  9. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 43
  10. 2024Can Knowledge Editing Really Correct Hallucinations?cited by 28
  11. 2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 28
  12. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 27
38 more
  1. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  2. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judgescited by 24
  3. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 24
  4. 2024Can Editing LLMs Inject Harm?cited by 22
  5. 2025BingoGuard: LLM Content Moderation Tools with Risk Levelscited by 21
  6. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  7. 2024WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Modelscited by 17
  8. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 17
  9. 2025Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Modelscited by 15
  10. 2024Risks and Opportunities of Open-Source Generative AIcited by 13
  11. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Modelscited by 12
  12. 2026Automated Benchmark Auditing for AI Agents and Large Language Modelscited by 12
  13. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecturecited by 11
  14. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  15. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  16. 2025VLDBench Evaluating Multimodal Disinformation with Regulatory Alignmentcited by 9
  17. 2025HoliSafe: Holistic Safety Benchmarking and Modeling for Vision-Language Modelcited by 9
  18. 2025RealHarm: A Collection of Real-World Language Model Application Failurescited by 8
  19. 2024A Standardized Machine-readable Dataset Documentation Format for Responsible AIcited by 7
  20. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 7
  21. 2026When Should We Introduce Safety Interventions During Pretraining?cited by 7
  22. 2024SocialGaze: Improving the Integration of Human Social Norms in Large Language Modelscited by 4
  23. 2024Sensitive Content Classification in Social Media: A Holistic Resource and Evaluationcited by 4
  24. 2025Mind the Gesture: Evaluating AI Sensitivity to Culturally Offensive Non-Verbal Gesturescited by 4
  25. 2025OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!cited by 4
  26. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  27. 2025A Different Approach to AI Safety: Proceedings from the Columbia Convening on Openness in Artificial Intelligence and AI Safetycited by 3
  28. 2025Risk Management for Mitigating Benchmark Failure Modes: BenchRiskcited by 3
  29. 2026Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safetycited by 3
  30. 2026Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agentscited by 3
  31. 2026Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safetycited by 3
  32. 2025Supervision policies can shape long-term risk management in general-purpose AI modelscited by 2
  33. 2024SoK: Towards Security and Safety of Edge AIcited by 1
  34. 2025Responsible AI Technical Reportcited by 1
  35. 2026TWGuard: A Case Study of LLM Safety Guardrails for Localized Linguistic Contextscited by 1
  36. 2024ToVo: Toxicity Taxonomy via Votingcited by 0
  37. 2025Taxonomy of Comprehensive Safety for Clinical Agentscited by 0
  38. 2025Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFTcited by 0

50 most cited of 62 All 62 (JSON)