mlsanityData to 14 Oct 2026

Uncovering Safety Risks of Large Language Models through Concept Activation Vector

arXiv 2404.12038 ↗First posted 18 Apr 2024cs.CLcited by 67

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Despite careful safety alignment, current large language models (LLMs) remain vulnerable to various attacks. To further unveil the safety risks of LLMs, we introduce a Safety Concept Activation Vector (SCAV) framework, which effectively guides the attacks by accurately interpreting LLMs' safety mechanisms. We then develop an SCAV-guided attack method that can generate both attack prompts and embedding-level attacks with automatically selected perturbation hyperparameters. Both automatic and human evaluations demonstrate that our attack method significantly improves the attack success rate and response quality while requiring less training data. Additionally, we find that our generated attack prompts may be transferable to GPT-4, and the embedding-level attacks may also be transferred to other white-box LLMs whose parameters are known. Our experiments further uncover the safety risks present in current LLMs. For example, in our evaluation of seven open-source LLMs, we observe an average attack success rate of 99.14%, based on the classic keyword-matching criterion. Finally, we provide insights into the safety mechanism of LLMs. The code is available at https://github.com/SproutNan/AI-Safety_SCAV.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 49 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

31 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  2. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  9. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  10. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  11. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  12. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
13 more
  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  2. 2024A StrongREJECT for Empty Jailbreakscited by 385
  3. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  4. 2023Large Language Model Unlearningcited by 302
  5. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  6. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  7. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  8. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  9. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  10. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 51
  11. 2022Self-explaining deep models with logic rule reasoningcited by 13
  12. 2023Semi-Offline Reinforcement Learning for Optimized Text Generationcited by 12
  13. 2024Unveiling the Misuse Potential of Base Large Language Models via In-Context Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 45; 2026: 21.

67 papers.

Sort:

By area:

  1. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  2. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 54
  3. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  4. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 40
  5. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 38
  6. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  7. 2025Representation Bending for Large Language Model Safetycited by 30
  8. 2025The Rogue Scalpel: Activation Steering Compromises LLM Safetycited by 21
  9. 2026ODESteer: A Unified ODE-Based Steering Framework for LLM Alignmentcited by 19
  10. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
  11. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 15
  12. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 13
38 more
  1. 2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 13
  2. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 12
  3. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  4. 2025Can LLMs Explain Themselves Counterfactually?cited by 10
  5. 2025JailBound: Jailbreaking Internal Safety Boundaries of Vision-Language Modelscited by 10
  6. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  7. 2025Probing then Editing Response Personality of Large Language Modelscited by 9
  8. 2025Investigating CoT Monitorability in Large Reasoning Modelscited by 9
  9. 2025Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Directioncited by 8
  10. 2025Probing the Difficulty Perception Mechanism of Large Language Modelscited by 8
  11. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  12. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
  13. 2025TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluationscited by 7
  14. 2026When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agentscited by 7
  15. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 6
  16. 2025Evaluate Bias without Manual Test Sets: A Concept Representation Perspective for LLMscited by 6
  17. 2025Safe-SAIL: Towards a Fine-grained Safety Landscape of Large Language Models via Sparse Autoencoder Interpretation Frameworkcited by 6
  18. 2025Bleeding Pathways: Vanishing Discriminability in LLM Hidden States Fuels Jailbreak Attackscited by 5
  19. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspectivecited by 5
  20. 2025CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representationscited by 5
  21. 2025Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encodingcited by 5
  22. 2026Steering at the Source: Style Modulation Heads for Robust Persona Controlcited by 5
  23. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attackscited by 4
  24. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbationscited by 4
  25. 2025RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaborationcited by 4
  26. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasingcited by 3
  27. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  28. 2025Leveraging Machine Unlearning for Cost-Efficient Preference Alignmentcited by 2
  29. 2025InverseScope: Scalable Activation Inversion for Interpreting Large Language Modelscited by 2
  30. 2025When Smiley Turns Hostile: Interpreting How Emojis Trigger LLMs' Toxicitycited by 2
  31. 2025ProtoTS: Learning Hierarchical Prototypes for Explainable Time Series Forecastingcited by 2
  32. 2026Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Modelscited by 2
  33. 2026Enhancing Safety of Large Language Models via Embedding Space Separationcited by 2
  34. 2025Activation-Guided Local Editing for Jailbreaking Attackscited by 1
  35. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  36. 2025Forewarned is Forearmed: Pre-Synthesizing Jailbreak-like Instructions to Enhance LLM Safety Guardrail to Potential Attackscited by 1
  37. 2025Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Modelscited by 1
  38. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety Targeted Embedding Exploit via Refinement
  2. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuning
  3. 2024Recent Advances in Attack and Defense Approaches of Large Language Models
  4. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  5. 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation
  6. 2025Evaluating LLMs Robustness in Less Resourced Languages with Proxy Models
  7. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning
  8. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security