mlsanityData to 7 Oct 2026

Robust LLM safeguarding via refusal feature adversarial training

arXiv 2409.20089 ↗First posted 30 Sep 2024cs.LG cs.CL cs.CRcited by 60

Not on the map. Why?

Cited by 35 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are vulnerable to adversarial attacks that can elicit harmful responses. Defending against such attacks remains challenging due to the opacity of jailbreaking mechanisms and the high computational cost of training LLMs robustly. We demonstrate that adversarial attacks share a universal mechanism for circumventing LLM safeguards that works by ablating a dimension in the residual stream embedding space called the refusal feature. We further show that the operation of refusal feature ablation (RFA) approximates the worst-case perturbation of offsetting model safety. Based on these findings, we propose Refusal Feature Adversarial Training (ReFAT), a novel algorithm that efficiently performs LLM adversarial training by simulating the effect of input-level attacks via RFA. Experiment results show that ReFAT significantly improves the robustness of three popular LLMs against a wide range of adversarial attacks, with considerably less computational overhead compared to existing adversarial training methods.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

44 of 51 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2023Mistral 7Bcited by 3,065
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  12. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
32 more
  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2022Toy Models of Superpositioncited by 928
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  5. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  11. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  12. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  13. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  14. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  15. 2024A StrongREJECT for Empty Jailbreakscited by 352
  16. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  17. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  18. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  19. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  20. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  21. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  22. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  23. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
  24. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  25. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  26. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 91
  27. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  28. 2023Adversarial Attacks and Defenses in Large Language Models: Old and New Threatscited by 49
  29. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
  30. 2024WordGame: Efficient & Effective LLM Jailbreak via Simultaneous Obfuscation in Query and Responsecited by 28
  31. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  32. 2024Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methodscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 32; 2026: 26.

60 papers.

Sort:

By area:

  1. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  2. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  3. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  4. 2025Embodied AI Agents: Modeling the Worldcited by 57
  5. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  6. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  7. 2025Representation Bending for Large Language Model Safetycited by 28
  8. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  9. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  10. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  11. 2026Intelligent AI Delegationcited by 16
  12. 2026A Framework for Formalizing LLM Agent Securitycited by 16
38 more
  1. 2025COSMIC: Generalized Refusal Direction Identification in LLM Activationscited by 15
  2. 2025Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoningcited by 15
  3. 2025Short-length Adversarial Training Helps LLMs Defend Long-length Jailbreak Attacks: Theoretical and Empirical Evidencecited by 13
  4. 2026DynaWeb: Model-Based Reinforcement Learning of Web Agentscited by 12
  5. 2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
  6. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  7. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  8. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 10
  9. 2025Guiding Giants: Lightweight Controllers for Weighted Activation Steering in LLMscited by 8
  10. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  11. 2025Improving Large Language Model Safety with Contrastive Representation Learningcited by 7
  12. 2025BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Modelscited by 7
  13. 2026What Drives Representation Steering? A Mechanistic Case Study on Steering Refusalcited by 7
  14. 2025Assessing and Enhancing the Robustness of LLM-based Multi-Agent Systems Through Chaos Engineeringcited by 6
  15. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  16. 2025Hallucination reduction with CASAL: Contrastive Activation Steering For Amortized Learningcited by 6
  17. 2026Efficient and accurate steering of Large Language Models through attention-guided feature learningcited by 6
  18. 2024Generative Semantic Communication: Architectures, Technologies, and Applicationscited by 5
  19. 2025Adversarial Suffix Filtering: a Defense Pipeline for LLMscited by 5
  20. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  21. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  22. 2024Jailbreaking LLM-Controlled Robotscited by 4
  23. 2025Confidence Elicitation: A New Attack Vector for Large Language Modelscited by 4
  24. 2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 3
  25. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  26. 2026Fail-Closed Alignment for Large Language Modelscited by 2
  27. 2025ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attackcited by 1
  28. 2025MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMscited by 1
  29. 2026TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tamperingcited by 1
  30. 2026Efficient Refusal Ablation in LLM through Optimal Transportcited by 1
  31. 2026Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Headscited by 1
  32. 2026Understanding and Improving Continuous Adversarial Training for LLMs via In-context Learning Theorycited by 1
  33. 2026Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometrycited by 1
  34. 2026RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMscited by 1
  35. 2026Where Pretraining writes and Alignment reads: the asymmetry of Transformer weight spacecited by 1
  36. 2026Does Fine-Tuning Undo Activation Steering? Behavioural Recovery Without Weight-Edit Reversalcited by 1
  37. 2025RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositoriescited by 0
  38. 2026Closing the Distribution Gap in Adversarial Training for LLMscited by 0

50 most cited of 60 All 60 (JSON)