mlsanityData to 14 Oct 2026

Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs

arXiv 2308.13387 ↗First posted 25 Aug 2023cs.CLcited by 184

Not on the map. Why?

Cited by 73 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the rapid evolution of large language models (LLMs), new and hard-to-predict harmful capabilities are emerging. This requires developers to be able to identify risks through the evaluation of "dangerous capabilities" in order to responsibly deploy LLMs. In this work, we collect the first open-source dataset to evaluate safeguards in LLMs, and deploy safer open-source LLMs at a low cost. Our dataset is curated and filtered to consist only of instructions that responsible language models should not follow. We annotate and assess the responses of six popular LLMs to these instructions. Based on our annotation, we proceed to train several BERT-like classifiers, and find that these small classifiers can achieve results that are comparable with GPT-4 on automatic safety evaluation. Warning: this paper contains example data that may be offensive, harmful, or biased.

Key facts. Cited by 184 papers in the map (data through 14 Oct 2026). Builds on 21 of its 29 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

21 of 29 references.

References · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  11. 2020Recipes for building an open-domain chatbotcited by 593
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
9 more
  1. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
  2. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  3. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  4. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  5. 2023Model evaluation for extreme riskscited by 177
  6. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 148
  7. 2021Evaluating Debiasing Techniques for Intersectional Biasescited by 30
  8. 2023Fair Enough: Standardizing Evaluation and Model Selection for Fairness Research in NLPcited by 11
  9. 2022Systematic Evaluation of Predictive Fairnesscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 71; 2025: 49; 2026: 52.

184 papers.

Sort:

By area:

  1. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  4. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  6. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  7. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  8. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  9. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  10. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  11. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  12. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
38 more
  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 176
  2. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  3. 2025Large Language Models Often Know When They Are Being Evaluatedcited by 128
  4. 2023Factuality Challenges in the Era of Large Language Modelscited by 126
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  6. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  7. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 94
  8. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
  9. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 77
  10. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 74
  11. 2023Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Modelscited by 69
  12. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 68
  13. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  14. 2024Building Guardrails for Large Language Modelscited by 65
  15. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 60
  16. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Modelscited by 53
  17. 2024CoSafe: Evaluating Large Language Model Safety in Multi-Turn Dialogue Coreferencecited by 51
  18. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 47
  19. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 47
  20. 2024Improving Reward Models with Synthetic Critiquescited by 44
  21. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoningcited by 43
  22. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  23. 2023Flames: Benchmarking Value Alignment of LLMs in Chinesecited by 37
  24. 2024Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Modelscited by 37
  25. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  26. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 34
  27. 2024S-Eval: Towards Automated and Comprehensive Safety Evaluation for Large Language Modelscited by 33
  28. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
  29. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  30. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 27
  31. 2024Exploring Safety-Utility Trade-Offs in Personalized Language Modelscited by 27
  32. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  33. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  34. 2023Disinformation Capabilities of Large Language Modelscited by 25
  35. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 24
  36. 2024Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metriccited by 23
  37. 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responsescited by 23
  38. 2024LLM-Driven Robots Risk Enacting Discrimination, Violence, and Unlawful Actionscited by 22

50 most cited of 184 All 184 (JSON)