mlsanityData to 14 Oct 2026

Are aligned neural networks adversarially aligned?

arXiv 2306.15447 ↗First posted 26 Jun 2023cs.CL cs.AI cs.CRcited by 315

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Large language models are now tuned to align with the goals of their creators, namely to be "helpful and harmless." These models should respond helpfully to user questions, but refuse to answer requests that could cause harm. However, adversarial users can construct inputs which circumvent attempts at alignment. In this work, we study adversarial alignment, and ask to what extent these models remain aligned when interacting with an adversarial user who constructs worst-case inputs (adversarial examples). These inputs are designed to cause the model to emit harmful content that would otherwise be prohibited. We show that existing NLP-based optimization attacks are insufficiently powerful to reliably attack aligned text models: even when current NLP-based attacks fail, we can find adversarial inputs with brute force. As a result, the failure of current attacks should not be seen as proof that aligned text models remain aligned under adversarial inputs. However the recent trend in large-scale ML models is multimodal models that allow users to provide images that influence the text that is generated. We show these models can be easily attacked, i.e., induced to perform arbitrary un-aligned behavior through adversarial perturbation of the input image. We conjecture that improved NLP attacks may demonstrate this same level of adversarial control over text-only models.

Key facts. Cited by 315 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 57 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

39 of 57 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2013Intriguing properties of neural networkscited by 5,155
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  11. 2017Deep reinforcement learning from human preferencescited by 3,912
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
27 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2022Emergent Abilities of Large Language Modelscited by 2,512
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2023PaLM 2 Technical Reportcited by 1,162
  5. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 744
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  8. 2017Evasion Attacks against Machine Learning at Test Timecited by 689
  9. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  10. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  11. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 567
  12. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  13. 2018Generating Natural Language Adversarial Examplescited by 424
  14. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  15. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
  16. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  17. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  18. 2022Predictability and Surprise in Large Generative Modelscited by 210
  19. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  20. 2021Challenges in Detoxifying Language Modelscited by 169
  21. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 165
  22. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  23. 2018Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executablescited by 62
  24. 2018AdVersarial: Perceptual Ad Blocking meets Adversarial Machine Learningcited by 14
  25. 2022Current and Near-Term AI as a Potential Existential Risk Factorcited by 14
  26. 2022Increasing Confidence in Adversarial Robustness Evaluationscited by 12
  27. 2023Randomness in ML Defenses Helps Persistent Attackers and Hinders Evaluatorscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 45; 2024: 126; 2025: 86; 2026: 58.

315 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
  8. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  9. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  10. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  12. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
38 more
  1. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  2. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  3. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 398
  4. 2023Explainability for Large Language Models: A Surveycited by 333
  5. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  6. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  7. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  8. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 273
  9. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 244
  10. 2023Large Language Model Alignment: A Surveycited by 243
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  12. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  13. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  14. 2023How Robust is Google's Bard to Adversarial Image Attacks?cited by 183
  15. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  16. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  17. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
  18. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  19. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  20. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  21. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  22. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  23. 2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
  24. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  25. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 136
  26. 2024Jailbreaking Attack against Multimodal Large Language Modelcited by 135
  27. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  28. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  29. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 128
  30. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  31. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  32. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 124
  33. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  34. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  35. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  36. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 109
  37. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  38. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95

50 most cited of 315 All 315 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Adversarial Attacks on Large Language Models Using Regularized Relaxation
  2. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
  3. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  4. 2026Adversarial Alignment: Ensuring Value Consistency in Large Language Models for Sensitive Domains
  5. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbations
  6. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  7. 2025Rethinking Deep Alignment Through The Lens Of Incomplete Learning
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models