mlsanityData to 7 Oct 2026

AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment

arXiv 2410.11283 ↗First posted 15 Oct 2024cs.LGcited by 5

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

With the growing adoption of reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs), the risk of backdoor installation during alignment has increased, leading to unintended and harmful behaviors. Existing backdoor triggers are typically limited to fixed word patterns, making them detectable during data cleaning and easily removable post-poisoning. In this work, we explore the use of prompt-specific paraphrases as backdoor triggers, enhancing their stealth and resistance to removal during LLM alignment. We propose AdvBDGen, an adversarially fortified generative fine-tuning framework that automatically generates prompt-specific backdoors that are effective, stealthy, and transferable across models. AdvBDGen employs a generator-discriminator pair, fortified by an adversary, to ensure the installability and stealthiness of backdoors. It enables the crafting and successful installation of complex triggers using as little as 3% of the fine-tuning data. Once installed, these backdoors can jailbreak LLMs during inference, demonstrate improved stability against perturbations compared to traditional constant triggers, and are more challenging to remove. These findings underscore an urgent need for the research community to develop more robust defenses against adversarial backdoor threats in LLM alignment.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 51 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

38 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think17 Jun
  2. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  7. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  8. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  9. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  10. 2018Spectral Signatures in Backdoor Attackscited by 457
  11. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
20 more
  1. 2020Invisible Backdoor Attack with Sample-Specific Triggerscited by 338
  2. 2023Poisoning Language Models During Instruction Tuningcited by 254
  3. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  4. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  5. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  6. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  7. 2021Adversarial Unlearning of Backdoors via Implicit Hypergradientcited by 160
  8. 2017Neural Trojanscited by 151
  9. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  10. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  11. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  12. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  13. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitutioncited by 85
  14. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  15. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  16. 2022Not All Poisons are Created Equal: Robust Training against Data Poisoningcited by 29
  17. 2020ConFoc: Content-Focus Protection Against Trojan Attacks on Neural Networkscited by 28
  18. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27
  19. 2021Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoningcited by 19
  20. 2024Round Trip Translation Defence against Large Language Model Jailbreaking Attackscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignmentcited by 3
  2. 2025RAGPart & RAGMask: Retrieval-Stage Defenses Against Corpus Poisoning in Retrieval-Augmented Generationcited by 3
  3. 2025Teach a Reward Model to Correct Itself: Reward Guided Adversarial Failure Discovery for Robust Reward Modelingcited by 2
  4. 2026Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base modelcited by 1
  5. 2026Compliance2LoRA: Personalizable On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapterscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
  2. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
  3. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challenges
  4. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
  5. 2025HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
  6. 2025Formalization Driven LLM Prompt Jailbreaking via Reinforcement Learning
  7. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  8. 2025AutoBackdoor: Automating Backdoor Attacks via LLM Agents