mlsanityData to 14 Oct 2026

SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety

arXiv 2404.05399 ↗First posted 8 Apr 2024cs.CL cs.AIcited by 77

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

The last two years have seen a rapid growth in concerns around the safety of large language models (LLMs). Researchers and practitioners have met these concerns by creating an abundance of datasets for evaluating and improving LLM safety. However, much of this work has happened in parallel, and with very different goals in mind, ranging from the mitigation of near-term risks around bias and toxic content generation to the assessment of longer-term catastrophic risk potential. This makes it difficult for researchers and practitioners to find the most relevant datasets for their use case, and to identify gaps in dataset coverage that future work may fill. To remedy these issues, we conduct a first systematic review of open datasets for evaluating and improving LLM safety. We review 144 datasets, which we identified through an iterative and community-driven process over the course of several months. We highlight patterns and trends, such as a trend towards fully synthetic datasets, as well as gaps in dataset coverage, such as a clear lack of non-English and naturalistic datasets. We also examine how LLM safety datasets are used in practice -- in LLM release publications and popular LLM benchmarks -- finding that current evaluation practices are highly idiosyncratic and make use of only a small fraction of available datasets. Our contributions are based on SafetyPrompts.com, a living catalogue of open datasets for LLM safety, which we plan to update continuously as the field of LLM safety develops.

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 87 of its 126 references that are in the map. First posted on arXiv 8 Apr 2024.

Builds on

87 of 126 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming6 Apr
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 85

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2023Mistral 7Bcited by 3,164
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
38 more
  1. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  2. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  3. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  6. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  7. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2023Whose Opinions Do Language Models Reflect?cited by 773
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2020Aligning AI With Shared Human Valuescited by 723
  12. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  13. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  14. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  15. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  16. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  17. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  18. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  19. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  20. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  21. 2018Gender Bias in Coreference Resolutioncited by 442
  22. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  23. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  24. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  25. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  26. 2024A StrongREJECT for Empty Jailbreakscited by 385
  27. 2023Are aligned neural networks adversarially aligned?cited by 315
  28. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  29. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 285
  30. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  31. 2020Social Chemistry 101: Learning to Reason about Social and Moral Normscited by 255
  32. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
  33. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  34. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
  35. 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 226
  36. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  37. 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
  38. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 33; 2026: 23.

77 papers.

Sort:

By area:

  1. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  4. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  5. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  6. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  7. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  8. 2025Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluationcited by 60
  9. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 45
  10. 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 37
  11. 2024Towards interactive evaluations for interaction harms in human-AI systemscited by 29
  12. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
38 more
  1. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 27
  2. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  3. 2024LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actionscited by 22
  4. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 22
  5. 2025Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMscited by 20
  6. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 18
  7. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agentscited by 18
  8. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  9. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 16
  10. 2024ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domaincited by 15
  11. 2025MSTS: A Multimodal Safety Test Suite for Vision-Language Modelscited by 12
  12. 2024How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studiescited by 10
  13. 2024SEAS: Self-Evolving Adversarial Safety Optimization for Large Language Modelscited by 9
  14. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMscited by 9
  15. 2024Blockchain for Large Language Model Security and Safety: A Holistic Surveycited by 8
  16. 2025Comprehensive Analysis of Transparency and Accessibility of ChatGPT, DeepSeek, And other SoTA Large Language Modelscited by 8
  17. 2025HuggingGraph: Understanding the Supply Chain of LLM Ecosystemcited by 8
  18. 2025Benchmarking is Broken -- Don't Let AI be its Own Judgecited by 8
  19. 2024Decoding Hate: Exploring Language Models' Reactions to Hate Speechcited by 6
  20. 2024Unanswerability Evaluation for Retrieval Augmented Generationcited by 4
  21. 2025EuroGEST: Investigating gender stereotypes in multilingual language modelscited by 4
  22. 2025Towards Safer AI Moderation: Evaluating LLM Moderators Through a Unified Benchmark Dataset and Advocating a Human-First Approachcited by 4
  23. 2025SAFENLIDB: A Privacy-Preserving Safety Alignment Framework for LLM-based Natural Language Database Interfacescited by 4
  24. 2026StealthGraph: Exposing Domain-Specific Risks in LLMs through Knowledge-Graph-Guided Harmful Prompt Generationcited by 4
  25. 2024Trust & Safety of LLMs and LLMs in Trust & Safetycited by 3
  26. 2025Societal Impacts Research Requires Benchmarks for Creative Composition Taskscited by 3
  27. 2025Risk Management for Mitigating Benchmark Failure Modes: BenchRiskcited by 3
  28. 2026MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Modelscited by 3
  29. 2024Compromesso! Italian Many-Shot Jailbreaks Undermine the Safety of Large Language Modelscited by 2
  30. 2024A Systematic Review of NeurIPS Dataset Management Practicescited by 2
  31. 2025A Systematic Review of Open Datasets Used in Text-to-Image (T2I) Gen AI Model Safetycited by 2
  32. 2025The Structural Safety Generalization Problemcited by 2
  33. 2025SCAN: Structured Capability Assessment and Navigation for LLMscited by 2
  34. 2025Evaluating Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguardscited by 2
  35. 2025Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gapscited by 2
  36. 2025Tailored untruths: How personalisation challenges LLM safeguardscited by 2
  37. 2025Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Modelscited by 2
  38. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Models
  2. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
  3. 2026SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond
  4. 2023All Languages Matter: On the Multilingual Safety of Large Language Models
  5. 2024SafeBench: A Safety Evaluation Framework for Multimodal Large Language Models
  6. 2025A Data-Centric Approach for Safe and Secure Large Language Models against Threatening and Toxic Content
  7. 2026IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia
  8. 2025LLM-Safety Evaluations Lack Robustness