Data to 14 Oct 2026

Defending Against Unforeseen Failure Modes with Latent Adversarial Training

arXiv 2403.05030 ↗First posted 8 Mar 2024cs.CR cs.AI cs.LGcited by 88

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Despite extensive diagnostics and debugging by developers, AI systems sometimes exhibit harmful unintended behaviors. Finding and fixing these is challenging because the attack surface is so large -- it is not tractable to exhaustively search for inputs that may elicit harmful behaviors. Red-teaming and adversarial training (AT) are commonly used to improve robustness, however, they empirically struggle to fix failure modes that differ from the attacks used during training. In this work, we utilize latent adversarial training (LAT) to defend against vulnerabilities without leveraging knowledge of what they are or using inputs that elicit them. LAT makes use of the compressed, abstract, and structured latent representations of concepts that the network actually uses for prediction. Here, we use it to defend against failure modes without examples that elicit them. Specifically, we use LAT to remove backdoors and defend against held-out classes of adversarial attacks. We show in image classification, text classification, and text generation tasks that LAT usually improves both robustness to novel attacks and performance on clean data relative to AT. This suggests that LAT can be a promising tool for defending against failure modes that are not explicitly identified by developers.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 84 of its 125 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

84 of 125 references.

Closest ancestors · 1

Same direction, newest first

2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 83

2015Deep Residual Learning for Image Recognitioncited by 52,096
2023GPT-4 Technical Reportcited by 18,396
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
2014Explaining and Harnessing Adversarial Examplescited by 7,364
2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
2013Intriguing properties of neural networkscited by 5,155
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
2020Shortcut Learning in Deep Neural Networkscited by 1,969
2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850

38 more

2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
2019Natural Adversarial Examplescited by 1,408
2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
2018Robustness May Be at Odds with Accuracycited by 880
2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
2022Quantifying Memorization Across Neural Language Modelscited by 763
2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
2023Detecting Pretraining Data from Large Language Modelscited by 387
2023Are aligned neural networks adversarially aligned?cited by 315
2023Low-Resource Languages Jailbreak GPT-4cited by 309
2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
2021Unsolved Problems in ML Safetycited by 292
2023Poisoning Web-Scale Training Datasets is Practicalcited by 290
2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 264
2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 254
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 172
2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
2022Fine-tuned Language Models are Continual Learnerscited by 156
2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
2023Understanding Catastrophic Forgetting in Language Models via Implicit Inferencecited by 144
2020Perceptual Adversarial Robustness: Defense Against Unseen Threat Modelscited by 137
2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 128
2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 115

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 35; 2026: 34.

88 papers.

Sort:

By area:

2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 113
2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 41
2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 37

38 more

2024Erasing Conceptual Knowledge from Language Modelscited by 34
2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 34
2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
2024Games for AI Control: Models of Safety Evaluations of AI Deployment Protocolscited by 31
2024Failures to Find Transferable Image Jailbreaks Between Vision-Language Modelscited by 30
2024Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Modelscited by 27
2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustnesscited by 20
2024The Potential and Challenges of Evaluating Attitudes, Opinions, and Values in Large Language Modelscited by 19
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 16
2025Consistency Training Helps Stop Sycophancy and Jailbreakscited by 16
2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 15
2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 14
2025STACK: Adversarial Attacks on LLM Safeguard Pipelinescited by 14
2025Learning Safety Constraints for Large Language Modelscited by 13
2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 13
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2025A Survey of Theory of Mind in Large Language Models: Evaluations, Representations, and Safety Riskscited by 8
2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 8
2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7
2025Adversarial D'ej`a Vu: Jailbreak Dictionary Learning for Stronger Generalization to Unseen Attackscited by 7
2026Interpretability Can Be Actionablecited by 7
2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 6
2025Backdoor Collapse: Eliminating Unknown Threats via Known Backdoor Aggregation in Language Modelscited by 6
2026Sparse Autoencoders are Capable LLM Jailbreak Mitigatorscited by 6
2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
2025Same Question, Different Words: A Latent Adversarial Framework for Prompt Robustnesscited by 5
2024Unelicitable Backdoors in Language Models via Cryptographic Transformer Circuitscited by 4

50 most cited of 88 All 88 (JSON)

Similar, not cited

Close in wording, no citation either way