mlsanityData to 7 Oct 2026

Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs

arXiv 2407.15549 ↗First posted 22 Jul 2024cs.LG cs.AI cs.CLcited by 158

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) can often be made to behave in undesirable ways that they are explicitly fine-tuned not to. For example, the LLM red-teaming literature has produced a wide variety of 'jailbreaking' techniques to elicit harmful text from models that were fine-tuned to be harmless. Recent work on red-teaming, model editing, and interpretability suggests that this challenge stems from how (adversarial) fine-tuning largely serves to suppress rather than remove undesirable capabilities from LLMs. Prior work has introduced latent adversarial training (LAT) as a way to improve robustness to broad classes of failures. These prior works have considered untargeted latent space attacks where the adversary perturbs latent activations to maximize loss on examples of desirable behavior. Untargeted LAT can provide a generic type of robustness but does not leverage information about specific failure modes. Here, we experiment with targeted LAT where the adversary seeks to minimize loss on a specific competing task. We find that it can augment a wide variety of state-of-the-art methods. First, we use targeted LAT to improve robustness to jailbreaks, outperforming a strong R2D2 baseline with orders of magnitude less compute. Second, we use it to more effectively remove backdoors with no knowledge of the trigger. Finally, we use it to more effectively unlearn knowledge for specific undesirable tasks in a way that is also more robust to re-learning. Overall, our results suggest that targeted LAT can be an effective tool for defending against harmful behaviors from LLMs.

Key facts. Cited by 158 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 113 of its 139 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

113 of 139 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024Efficient Adversarial Training in LLMs with Continuous Attacks24 May
  2. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMs22 Apr
  3. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledge8 Apr
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  5. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models26 Mar
  6. 2024Coercing LLMs to do and reveal (almost) anything21 Feb
  7. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  8. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  9. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Clues14 Feb
  10. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  11. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  12. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
8 more
  1. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  2. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  3. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks16 Oct
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  5. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 93

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2023Mistral 7Bcited by 3,065
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  13. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  14. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  15. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  16. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  17. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  18. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  19. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  20. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  21. 2023Are aligned neural networks adversarially aligned?cited by 305
  22. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  23. 2023Large Language Model Unlearningcited by 288
  24. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  25. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  26. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  27. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  28. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  29. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  30. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  31. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  32. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  33. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  34. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  35. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  36. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  37. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  38. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 77; 2026: 64.

158 papers.

Sort:

By area:

  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  2. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  3. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  4. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  5. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  6. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  7. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  8. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  9. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  10. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  11. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  12. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
38 more
  1. 2025One-shot Optimized Steering Vectors Mediate Safety-relevant Behaviors in LLMscited by 27
  2. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  3. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  4. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  5. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  6. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 20
  7. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  8. 2024Endless Jailbreaks with Bijection Learningcited by 18
  9. 2024Bayesian scaling laws for in-context learningcited by 18
  10. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  11. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  12. 2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 15
  13. 2025The Elicitation Game: Evaluating Capability Elicitation Techniquescited by 15
  14. 2025Multi-Turn Jailbreaks Are Simpler Than They Seemcited by 15
  15. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  16. 2025Safety Reasoning with Guidelinescited by 14
  17. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
  18. 2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
  19. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  20. 2025Lifelong Safety Alignment for Language Modelscited by 13
  21. 2025Learning Safety Constraints for Large Language Modelscited by 13
  22. 2025The dark deep side of DeepSeek: Fine-tuning attacks against the safety alignment of CoT-enabled modelscited by 12
  23. 2025Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Trainingcited by 12
  24. 2025Distillation Robustifies Unlearningcited by 11
  25. 2026Shaping capabilities with token-level data filteringcited by 11
  26. 2025The Steganographic Potentials of Language Modelscited by 10
  27. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  28. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 9
  29. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  30. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  31. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  32. 2024Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vectorcited by 8
  33. 2025Jailbreaking to Jailbreakcited by 8
  34. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
  35. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  36. 2025UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearningcited by 7
  37. 2025LLM Fingerprinting via Semantically Conditioned Watermarkscited by 7
  38. 2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7

50 most cited of 158 All 158 (JSON)