Data to 7 Oct 2026

Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space

arXiv 2402.09063 ↗First posted 14 Feb 2024cs.LGcited by 88

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more

LLM safety and security → Jailbreaking

Abstract (arXiv)

Current research in adversarial robustness of LLMs focuses on discrete input manipulations in the natural language space, which can be directly transferred to closed-source models. However, this approach neglects the steady progression of open-source models. As open-source models advance in capability, ensuring their safety also becomes increasingly imperative. Yet, attacks tailored to open-source LLMs that exploit full model access remain largely unexplored. We address this research gap and propose the embedding space attack, which directly attacks the continuous embedding representation of input tokens. We find that embedding space attacks circumvent model alignments and trigger harmful behaviors more efficiently than discrete attacks or model fine-tuning. Furthermore, we present a novel threat model in the context of unlearning and show that embedding space attacks can extract supposedly deleted information from unlearned LLMs across multiple datasets and models. Our findings highlight embedding space attacks as an important threat model in open-source LLMs. Trigger Warning: the appendix contains LLM-generated text with violence and harassment.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 53 of its 83 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

53 of 83 references.

Closest ancestors · 7

Same direction, newest first

Other references · 46

2020Language Models are Few-Shot Learnerscited by 32,199
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2014Explaining and Harnessing Adversarial Examplescited by 7,287
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
2023Mistral 7Bcited by 3,065
2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
2020Extracting Training Data from Large Language Modelscited by 2,144
2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124

34 more

2016Universal adversarial perturbationscited by 774
2019Designing and Interpreting Probes with Control Taskscited by 608
2018Deep learning for time series classification: a reviewcited by 575
2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
2018Adversarial Attacks on Neural Networks for Graph Datacited by 395
2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
2023Detecting Pretraining Data from Large Language Modelscited by 375
2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
2020Adversarial Training for Large Neural Language Modelscited by 109
2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
2023Ewald-based Long-Range Message Passing for Molecular Graphscited by 46
2023Can Language Models be Instructed to Protect Personal Information?cited by 46
2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
2021Ab-Initio Potential Energy Surfaces by Pairing GNNs with Neural Wave Functionscited by 36
2021Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attackscited by 33
2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
2021CLIP: Cheap Lipschitz Training of Neural Networkscited by 29
2022Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantificationcited by 16
2020Time Matters: Time-Aware LSTMs for Predictive Business Process Monitoringcited by 14
2021Identifying Untrustworthy Predictions in Neural Networks by Geometric Gradient Analysiscited by 13
2023Hierarchical Randomized Smoothingcited by 9
2023Raising the Bar for Certified Adversarial Robustness with Diffusion Modelscited by 7
2020Dynamically Sampled Nonlocal Gradients for Stronger Adversarial Attackscited by 6
2023Provable Adversarial Robustness for Group Equivariant Tasks: Graphs, Point Clouds, Molecules, and Morecited by 3
2021System Design for a Data-driven and Explainable Customer Sentiment Monitorcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 45; 2026: 20.

88 papers.

Sort:

By area:

2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
2024Improving Alignment and Robustness with Circuit Breakerscited by 302
2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 63
2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36

38 more

2024Threats, Attacks, and Defenses in Machine Unlearning: A Surveycited by 34
2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 33
2025Representation Bending for Large Language Model Safetycited by 28
2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
2024Extracting Unlearned Information from LLMs with Activation Steeringcited by 19
2024Bayesian scaling laws for in-context learningcited by 18
2025LLM-Safety Evaluations Lack Robustnesscited by 16
2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
2024Adversarial Attacks on Large Language Models Using Regularized Relaxationcited by 12
2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlapcited by 12
2025Robust Answers, Fragile Logic: Probing the Decoupling Hypothesis in LLM Reasoningcited by 11
2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
2025Robustness and Cybersecurity in the EU Artificial Intelligence Actcited by 10
2025A Representation Engineering Perspective on the Effectiveness of Multi-Turn Jailbreakscited by 10
2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
2025Agents Are All You Need for LLM Unlearningcited by 9
2025SoK: Machine Unlearning for Large Language Modelscited by 9
2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
2024Self-interpreting Adversarial Imagescited by 7
2025Does Machine Unlearning Truly Remove Knowledge?cited by 7
2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 7
2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
2025Unraveling LLM Jailbreaks Through Safety Knowledge Neuronscited by 7
2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
2024Adversarial Vulnerabilities in Large Language Models for Time Series Forecastingcited by 6
2025Harry Potter is Still Here! Probing Knowledge Leakage in Targeted Unlearned Large Language Models via Automated Adversarial Promptingcited by 6
2025Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Promptscited by 5
2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
2024Functional Homotopy: Smoothing Discrete Optimization via Continuous Parameters for LLM Jailbreak Attackscited by 4
2025SUA: Stealthy Multimodal Large Language Model Unlearning Attackcited by 4
2026REBEL: Hidden Knowledge Recovery via Evolutionary-Based Evaluation Loopcited by 4
2025Fast Proxies for LLM Robustness Evaluationcited by 3
2025Adversarial Attack on Large Language Models using Exponentiated Gradient Descentcited by 3

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way