mlsanityData to 7 Oct 2026

AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies

arXiv 2407.17436 ↗First posted 11 Jul 2024cs.CY cs.AIcited by 69

Not on the map. Why?

Cited by 21 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Foundation models (FMs) provide societal benefits but also amplify risks. Governments, companies, and researchers have proposed regulatory frameworks, acceptable use policies, and safety benchmarks in response. However, existing public benchmarks often define safety categories based on previous literature, intuitions, or common sense, leading to disjointed sets of categories for risks specified in recent regulations and policies, which makes it challenging to evaluate and compare FMs across these benchmarks. To bridge this gap, we introduce AIR-Bench 2024, the first AI safety benchmark aligned with emerging government regulations and company policies, following the regulation-based safety categories grounded in our AI risks study, AIR 2024. AIR 2024 decomposes 8 government regulations and 16 company policies into a four-tiered safety taxonomy with 314 granular risk categories in the lowest tier. AIR-Bench 2024 contains 5,694 diverse prompts spanning these categories, with manual curation and human auditing to ensure quality. We evaluate leading language models on AIR-Bench 2024, uncovering insights into their alignment with specified safety concerns. By bridging the gap between public benchmarks and practical AI risks, AIR-Bench 2024 provides a foundation for assessing model safety across jurisdictions, fostering the development of safer and more responsible AI systems.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Builds on 30 of its 63 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

30 of 63 references.

References · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2022Red Teaming Language Models with Language Modelscited by 983
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  12. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
18 more
  1. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  2. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  4. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  6. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  7. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  8. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  9. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  10. 2023An Overview of Catastrophic AI Riskscited by 214
  11. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  12. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  13. 2013Stabilitycited by 119
  14. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  15. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 80
  16. 2023Managing extreme AI risks amid rapid progresscited by 48
  17. 2024AI Risk Categorization Decoded (AIR 2024): From Government Regulations to Corporate Policiescited by 34
  18. 2023An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 36; 2026: 27.

69 papers.

Sort:

By area:

  1. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  2. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  3. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  4. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  5. 2025PolyGuard: A Multilingual Safety Moderation Tool for 17 Languagescited by 57
  6. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  7. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 38
  8. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  9. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 29
  10. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  11. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  12. 2025Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmascited by 17
38 more
  1. 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contaminationcited by 13
  2. 2024Acceptable Use Policies for Foundation Modelscited by 12
  3. 2025Guiding not Forcing: Enhancing the Transferability of Jailbreaking Attacks on LLMs via Removing Superfluous Constraintscited by 11
  4. 2024SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behaviorcited by 10
  5. 2025Fantastic Bugs and Where to Find Them in AI Benchmarkscited by 9
  6. 2025MedSentry: Understanding and Mitigating Safety Risks in Medical LLM Multi-Agent Systemscited by 8
  7. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  8. 2025Enhancing Lexicon-Based Text Embeddings with Large Language Modelscited by 6
  9. 2025Bench-2-CoP: Can We Trust Benchmarking for EU AI Compliance?cited by 6
  10. 2024Focus On This, Not That! Steering LLMs with Adaptive Feature Specificationcited by 5
  11. 2025EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistantscited by 5
  12. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  13. 2025Beyond Benchmarks: On The False Promise of AI Regulationcited by 4
  14. 2025SafeProtein: Red-Teaming Framework and Benchmark for Protein Foundation Modelscited by 4
  15. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  16. 2025GuardTrace-VL: Detecting Unsafe Multimodel Reasoning via Iterative Safety Supervisioncited by 4
  17. 2025Implicit Bias-Like Patterns in Reasoning Modelscited by 3
  18. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failurescited by 3
  19. 2025PAC Bench: Do Foundation Models Understand Prerequisites for Executing Manipulation Policies?cited by 3
  20. 2026ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AIcited by 3
  21. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safetycited by 3
  22. 2026SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generationcited by 3
  23. 2024Differentially Private Kernel Density Estimationcited by 2
  24. 2025Breaking Down Bias: On The Limits of Generalizable Pruning Strategiescited by 2
  25. 2025Safety Degradation in AI Agentscited by 2
  26. 2025Q-resafe: Assessing Safety Risks and Quantization-aware Safety Patching for Quantized Large Language Modelscited by 2
  27. 2025AI Harmonics: a human-centric and harms severity-adaptive AI risk assessment frameworkcited by 2
  28. 2025HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignmentcited by 2
  29. 2025Risk Management for Mitigating Benchmark Failure Modes: BenchRiskcited by 2
  30. 2025A Longitudinal Measurement of Privacy Policy Evolution for Large Language Modelscited by 2
  31. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Abilitycited by 2
  32. 2026CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generationcited by 2
  33. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  34. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarkingcited by 1
  35. 2025Feature-Guided SAE Steering for Refusal-Rate Control using Contrasting Promptscited by 1
  36. 2025From Symptoms to Systems: An Expert-Guided Approach to Understanding Risks of Generative AI for Eating Disorderscited by 1
  37. 2026Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testingcited by 1
  38. 2026SEED-SET: Scalable Evolving Experimental Design for System-level Ethical Testingcited by 1

50 most cited of 69 All 69 (JSON)