mlsanityData to 7 Oct 2026

Investigating Adversarial Trigger Transfer in Large Language Models

arXiv 2404.16020 ↗First posted 24 Apr 2024cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Recent work has developed optimization procedures to find token sequences, called adversarial triggers, which can elicit unsafe responses from aligned language models. These triggers are believed to be highly transferable, i.e., a trigger optimized on one model can jailbreak other models. In this paper, we concretely show that such adversarial triggers are not consistently transferable. We extensively investigate trigger transfer amongst 13 open models and observe poor and inconsistent transfer. Our experiments further reveal a significant difference in robustness to adversarial triggers between models Aligned by Preference Optimization (APO) and models Aligned by Fine-Tuning (AFT). We find that APO models are extremely hard to jailbreak even when the trigger is optimized directly on the model. On the other hand, while AFT models may appear safe on the surface, exhibiting refusals to a range of unsafe instructions, we show that they are highly susceptible to adversarial triggers. Lastly, we observe that most triggers optimized on AFT models also generalize to new unsafe instructions from five diverse domains, further emphasizing their vulnerability. Overall, our work highlights the need for more comprehensive safety evaluations for aligned language models.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 63 references that are in the map. First posted on arXiv 24 Apr 2024.

Builds on

50 of 63 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  2. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
30 more
  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  4. 2023LIMA: Less Is More for Alignmentcited by 1,080
  5. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  7. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  9. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  10. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  11. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  13. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  14. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  15. 2023Poisoning Language Models During Instruction Tuningcited by 254
  16. 2023The False Promise of Imitating Proprietary LLMscited by 254
  17. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  18. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  19. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  20. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  21. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  22. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  23. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  24. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  25. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  26. 2023On the Exploitability of Instruction Tuningcited by 110
  27. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  28. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  29. 2023On the Challenges of Using Black-Box APIs for Toxicity Evaluation in Researchcited by 44
  30. 2023Using In-Context Learning to Improve Dialogue Safetycited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 0; 2026: 2.

4 papers.

  1. 2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 27
  2. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroringcited by 10
  3. 2026On the Hardness of Junking LLMscited by 0
  4. 2026All you need to break LLMs are Black-Box, Adapting, Efficient, Transferable, Harmful, Applicable ... Attackscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  3. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbations
  4. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  5. 2026Target-free Latent Safety Alignment
  6. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  7. 2025Universal and Transferable Adversarial Attack on Large Language Models Using Exponentiated Gradient Descent
  8. 2025Safety Alignment Should Be Made More Than Just A Few Attention Heads