mlsanityData to 7 Oct 2026

SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Types

arXiv 2410.21965 ↗First posted 29 Oct 2024cs.CLcited by 58

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Ensuring the safety of large language model (LLM) applications is essential for developing trustworthy artificial intelligence. Current LLM safety benchmarks have two limitations. First, they focus solely on either discriminative or generative evaluation paradigms while ignoring their interconnection. Second, they rely on standardized inputs, overlooking the effects of widespread prompting techniques, such as system prompts, few-shot demonstrations, and chain-of-thought prompting. To overcome these issues, we developed SG-Bench, a novel benchmark to assess the generalization of LLM safety across various tasks and prompt types. This benchmark integrates both generative and discriminative evaluation tasks and includes extended data to examine the impact of prompt engineering and jailbreak on LLM safety. Our assessment of 3 advanced proprietary LLMs and 10 open-source LLMs with the benchmark reveals that most LLMs perform worse on discriminative tasks than generative ones, and are highly susceptible to prompts, indicating poor generalization in safety alignment. We also explain these findings quantitatively and qualitatively to provide insights for future research.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 36 of its 48 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

36 of 48 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
24 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  7. 2024InternLM2 Technical Reportcited by 391
  8. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 293
  9. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  10. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  11. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  12. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  13. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  14. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  15. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  16. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  17. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  18. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 57
  19. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 57
  20. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  21. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 39
  22. 2021Rethinking Supervised Pre-training for Better Downstream Transferringcited by 35
  23. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  24. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 29; 2026: 29.

58 papers.

Sort:

By area:

  1. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  3. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  5. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  6. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  7. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agentscited by 16
  8. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approachcited by 14
  9. 2025Evaluation Faking: Unveiling Observer Effects in Safety Evaluation of Frontier AI Systemscited by 12
  10. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  11. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
  12. 2025SafeVid: Toward Safety Aligned Video Large Multimodal Modelscited by 9
38 more
  1. 2025Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMscited by 8
  2. 2025SAGE-Eval: Evaluating LLMs for Systematic Generalizations of Safety Factscited by 8
  3. 2025Value Compass Benchmarks: A Platform for Fundamental and Validated Evaluation of LLMs Valuescited by 6
  4. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  5. 2025Efficient Safety Retrofitting Against Jailbreaking for LLMscited by 4
  6. 2025MTCMB: A Multi-Task Benchmark Framework for Evaluating LLMs on Knowledge, Reasoning, and Safety in Traditional Chinese Medicinecited by 4
  7. 2025Eliciting and Analyzing Emergent Misalignment in State-of-the-Art Large Language Modelscited by 4
  8. 2025The PacifAIst Benchmark:Would an Artificial Intelligence Choose to Sacrifice Itself for Human Safety?cited by 4
  9. 2025SafeMT: Multi-turn Safety for Multimodal Language Modelscited by 4
  10. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failurescited by 3
  11. 2025Executable Governance for AI: Translating Policies into Rules Using LLMscited by 3
  12. 2026RedBench: A Universal Dataset for Comprehensive Red Teaming of Large Language Modelscited by 3
  13. 2026SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systemscited by 3
  14. 2026Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safetycited by 3
  15. 2026FLIPS: Instance-Fingerprinting for LLMs via Pseudo-random Sequencescited by 3
  16. 2025Mapping Overlaps in Benchmarks through Perplexity in the Wildcited by 2
  17. 2025A Hierarchical Imprecise Probability Approach to Reliability Assessment of Large Language Modelscited by 2
  18. 2026InfiCoEvalChain: A Blockchain-Based Decentralized Framework for Collaborative LLM Evaluationcited by 2
  19. 2026CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generationcited by 2
  20. 2026When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraintscited by 2
  21. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  22. 2026Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Rolescited by 2
  23. 2025RRTL: Red Teaming Reasoning Large Language Models in Tool Learningcited by 1
  24. 2025TeleEval-OS: Performance evaluations of large language models for operations schedulingcited by 1
  25. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarkingcited by 1
  26. 2025AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teamingcited by 1
  27. 2025Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attackscited by 1
  28. 2026From Helpfulness to Toxic Proactivity: Diagnosing Behavioral Misalignment in LLM Agentscited by 1
  29. 2026Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testingcited by 1
  30. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 1
  31. 2026From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directionscited by 1
  32. 2026Risk-Adjusted Harm Scoring for Automated Red Teaming for LLMs in Financial Servicescited by 1
  33. 2026Evaluating Reliability Gaps in Large Language Model Safety via Repeated Prompt Samplingcited by 1
  34. 2026Benchmarking the Safety of Large Language Models for Robotic Health Attendant Controlcited by 1
  35. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  36. 2026FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teamingcited by 1
  37. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  38. 2026Safeguard-Conditioned Uplift: Measuring Utility-Risk Frontiers for Dual-Use Biology Assistantscited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Large Language Model Safety: A Holistic Survey
  2. 2024AI Safety in Generative AI Large Language Models: A Survey
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
  4. 2025LLM-Safety Evaluations Lack Robustness
  5. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  6. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion
  7. 2025aiXamine: Simplified LLM Safety and Security
  8. 2025SafeLawBench: Towards Safe Alignment of Large Language Models