AdversariaL attacK sAfety aLIgnment(ALKALI): Safeguarding LLMs through GRACE: Geometric Representation-Aware Contrastive Enhancement- Introducing Adversarial Vulnerability Quality Index (AVQI)
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
Adversarial threats against LLMs are escalating faster than current defenses can adapt. We expose a critical geometric blind spot in alignment: adversarial prompts exploit latent camouflage, embedding perilously close to the safe representation manifold while encoding unsafe intent thereby evading surface level defenses like Direct Preference Optimization (DPO), which remain blind to the latent geometry. We introduce ALKALI, the first rigorously curated adversarial benchmark and the most comprehensive to date spanning 9,000 prompts across three macro categories, six subtypes, and fifteen attack families. Evaluation of 21 leading LLMs reveals alarmingly high Attack Success Rates (ASRs) across both open and closed source models, exposing an underlying vulnerability we term latent camouflage, a structural blind spot where adversarial completions mimic the latent geometry of safe ones. To mitigate this vulnerability, we introduce GRACE - Geometric Representation Aware Contrastive Enhancement, an alignment framework coupling preference learning with latent space regularization. GRACE enforces two constraints: latent separation between safe and adversarial completions, and adversarial cohesion among unsafe and jailbreak behaviors. These operate over layerwise pooled embeddings guided by a learned attention profile, reshaping internal geometry without modifying the base model, and achieve up to 39% ASR reduction. Moreover, we introduce AVQI, a geometry aware metric that quantifies latent alignment failure via cluster separation and compactness. AVQI reveals when unsafe completions mimic the geometry of safe ones, offering a principled lens into how models internally encode safety. We make the code publicly available at https://anonymous.4open.science/r/alkali-B416/README.md.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 74 references that are in the map. First posted on arXiv 10 Jun 2025.
Builds on
27 of 74 references.
Closest ancestors · 2
Same direction, newest first
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
Other references · 25
- 2023GPT-4 Technical Reportcited by 18,396
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2023Mistral 7Bcited by 3,164
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
13 more
- 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
- 2023Certifying LLM Safety against Adversarial Promptingcited by 266
- 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
- 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
- 2023Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networkscited by 97
- 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
- 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 37
- 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 35
- 2023Jailbreaker in Jail: Moving Target Defense for Large Language Modelscited by 15
- 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
- 2024GOAt: Explaining Graph Neural Networks via Graph Output Attributioncited by 10
- 2024Goal Hijacking Attack on Large Language Models via Pseudo-Conversation Injectioncited by 4
- 2025Iterative Prompting with Persuasion Skills in Jailbreaking Large Language Modelscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
- 2024Adversarial Feature Alignment: Balancing Robustness and Accuracy in Deep Learning via Adversarial Training
- 2025Adversarial Preference Learning for Robust LLM Alignment
- 2025Probing the Robustness of Large Language Models Safety to Latent Perturbations
- 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space
- 2026Zero-Shot Embedding Drift Detection: A Lightweight Defense Against Prompt Injections in LLMs
- 2025Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training
- 2025Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment