# Unveiling Safety Vulnerabilities of Large Language Models

arXiv: https://arxiv.org/abs/2311.04124 · HTML: https://mlsanity.com/paper/2311.04124/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 45 references that are in the map.
- First posted on arXiv 7 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [Confident Learning: Estimating Uncertainty in Dataset Labels](https://mlsanity.com/paper/1911.00068/) (2019)
- [Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models](https://mlsanity.com/paper/2111.02840/) (2021)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [A New Generation of Perspective API: Efficient Multilingual Character-level Transformers](https://mlsanity.com/paper/2202.11176/) (2022)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [MM-LIMA: Less Is More for Alignment in Multi-Modal Datasets](https://mlsanity.com/paper/2308.12067/) (2023)
- [Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models](https://mlsanity.com/paper/2206.08325/) (2022)
- [Visual Instruction Tuning with Polite Flamingo](https://mlsanity.com/paper/2307.01003/) (2023)
- [Gaining Insights into Unrecognized User Utterances in Task-Oriented Dialog Systems](https://mlsanity.com/paper/2204.05158/) (2022)

## Built on by (most cited first, 27 of 27)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [STAR-1: Safer Alignment of Reasoning LLMs with 1K Data](https://mlsanity.com/paper/2504.01903/) (2025)
- [Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinations](https://mlsanity.com/paper/2404.10960/) (2024)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2504.18041/) (2025)
- [Arabic Dataset for LLM Safeguard Evaluation](https://mlsanity.com/paper/2410.17040/) (2024)
- [Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI](https://mlsanity.com/paper/2409.15398/) (2024)
- [Large Reasoning Models Are Autonomous Jailbreak Agents](https://mlsanity.com/paper/2508.04039/) (2025)
- [Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations](https://mlsanity.com/paper/2403.06009/) (2024)
- [Alignment Studio: Aligning Large Language Models to Particular Contextual Regulations](https://mlsanity.com/paper/2403.09704/) (2024)
- [Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs](https://mlsanity.com/paper/2502.15427/) (2025)
- [Large Language Models can be Strong Self-Detoxifiers](https://mlsanity.com/paper/2410.03818/) (2024)
- [IF-GUIDE: Influence Function-Guided Detoxification of LLMs](https://mlsanity.com/paper/2506.01790/) (2025)
- [When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models](https://mlsanity.com/paper/2510.21285/) (2025)
- [Security Steerability is All You Need](https://mlsanity.com/paper/2504.19521/) (2025)
- [Dialz: A Python Toolkit for Steering Vectors](https://mlsanity.com/paper/2505.06262/) (2025)
- [MetaAID 2.5: A Secure Framework for Developing Metaverse Applications via Large Language Models](https://mlsanity.com/paper/2312.14480/) (2023)
- [Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignment](https://mlsanity.com/paper/2406.11039/) (2024)
- [Being Kind Isn't Always Being Safe: Diagnosing Affective Hallucination in LLMs](https://mlsanity.com/paper/2508.16921/) (2025)
- [ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Concepts](https://mlsanity.com/paper/2508.16325/) (2025)
- [Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability](https://mlsanity.com/paper/2512.01848/) (2025)
- [Test-Time Detoxification without Training or Learning Anything](https://mlsanity.com/paper/2602.02498/) (2026)
- [Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Models](https://mlsanity.com/paper/2511.08484/) (2025)
- [Adversarial Distilled Retrieval-Augmented Guarding Model for Online Malicious Intent Detection](https://mlsanity.com/paper/2509.14622/) (2025)
- [ASTRA: Agentic Steerability and Risk Assessment Framework](https://mlsanity.com/paper/2511.18114/) (2025)
- [PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models](https://mlsanity.com/paper/2606.09697/) (2026)
- [Granite.Trust Policy Tools: Shareable, Actionable Policies for Generative AI Applications](https://mlsanity.com/paper/2608.23870/) (2026)

## Abstract (arXiv)

As large language models become more prevalent, their possible harmful or inappropriate responses are a cause for concern. This paper introduces a unique dataset containing adversarial examples in the form of questions, which we call AttaQ, designed to provoke such harmful or inappropriate responses. We assess the efficacy of our dataset by analyzing the vulnerabilities of various models when subjected to it. Additionally, we introduce a novel automatic approach for identifying and naming vulnerable semantic regions - input semantic areas for which the model is likely to produce harmful outputs. This is achieved through the application of specialized clustering techniques that consider both the semantic similarity of the input attacks and the harmfulness of the model's responses. Automatically identifying vulnerable semantic regions enhances the evaluation of model weaknesses, facilitating targeted improvements to its safety mechanisms and overall reliability.