mlsanityData to 13 Oct 2026

Improving Alignment and Robustness with Circuit Breakers

arXiv 2406.04313 ↗First posted 6 Jun 2024cs.LG cs.AI cs.CLcited by 318

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

AI systems can take harmful actions and are highly vulnerable to adversarial attacks. We present an approach, inspired by recent advances in representation engineering, that interrupts the models as they respond with harmful outputs with "circuit breakers." Existing techniques aimed at improving alignment, such as refusal training, are often bypassed. Techniques such as adversarial training try to plug these holes by countering specific attacks. As an alternative to refusal training and adversarial training, circuit-breaking directly controls the representations that are responsible for harmful outputs in the first place. Our technique can be applied to both text-only and multimodal language models to prevent the generation of harmful outputs without sacrificing utility -- even in the presence of powerful unseen attacks. Notably, while adversarial robustness in standalone image recognition remains an open challenge, circuit breakers allow the larger multimodal system to reliably withstand image "hijacks" that aim to produce harmful content. Finally, we extend our approach to AI agents, demonstrating considerable reductions in the rate of harmful actions when they are under attack. Our approach represents a significant step forward in the development of reliable safeguards to harmful behavior and adversarial attacks.

Key facts. Cited by 318 papers in the map (data through 13 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 55 of its 70 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

55 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning5 Mar

Other references · 54

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,379
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,360
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,053
38 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,871
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,197
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,064
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,637
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,405
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,404
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,286
  10. 2023AgentBench: Evaluating LLMs as Agentscited by 1,242
  11. 2022Editing Models with Task Arithmeticcited by 1,101
  12. 2022Red Teaming Language Models with Language Modelscited by 1,014
  13. 2022Mass-Editing Memory in a Transformercited by 990
  14. 2018Robustness May Be at Odds with Accuracycited by 878
  15. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 833
  16. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 777
  17. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 664
  18. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 647
  19. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 616
  20. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 565
  21. 2021Fast Model Editing at Scalecited by 493
  22. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 493
  23. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 483
  24. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 464
  25. 2023Detecting Language Model Attacks with Perplexitycited by 429
  26. 2023Augmented Language Models: a Surveycited by 403
  27. 2023Are aligned neural networks adversarially aligned?cited by 315
  28. 2023Low-Resource Languages Jailbreak GPT-4cited by 306
  29. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 272
  30. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 243
  31. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178
  32. 2023Image Hijacks: Adversarial Images can Control Generative Models at Runtimecited by 170
  33. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  34. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  35. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 145
  36. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  37. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 90
  38. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 36; 2025: 154; 2026: 128.

318 papers.

Sort:

By area:

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 422
  2. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 209
  3. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 169
  4. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  5. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  6. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 123
  7. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
  8. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 119
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 116
  10. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 114
  11. 2025SafeArena: Evaluating the Safety of Autonomous Web Agentscited by 99
  12. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
38 more
  1. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 90
  2. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 87
  3. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 85
  4. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  5. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 76
  6. 2024Best-of-N Jailbreakingcited by 75
  7. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 73
  8. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  9. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  10. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 66
  11. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 63
  12. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 62
  13. 2024Applying sparse autoencoders to unlearn knowledge in language modelscited by 58
  14. 2025Foot-In-The-Door: A Multi-turn Jailbreak for LLMscited by 54
  15. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 54
  16. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  17. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  18. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  19. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 48
  20. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 47
  21. 2026Building Production-Ready Probes For Geminicited by 44
  22. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 42
  23. 2025AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integrationcited by 41
  24. 2025Infrastructure for AI Agentscited by 40
  25. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 38
  26. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 38
  27. 2026Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreakscited by 38
  28. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 37
  29. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 35
  30. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 35
  31. 2025Safety Guardrails for LLM-Enabled Robotscited by 34
  32. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 33
  33. 2024Backtracking Improves Generation Safetycited by 32
  34. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  35. 2024From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attackscited by 29
  36. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signalscited by 29
  37. 2025Representation Bending for Large Language Model Safetycited by 29
  38. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28

50 most cited of 318 All 318 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Robust Alignment of Circuit Breakers
  2. 2026The Persistent Vulnerability of Aligned AI Systems
  3. 2023The Alignment Problem in Context
  4. 2025Seeing the Threat: Vulnerabilities in Vision-Language Models to Adversarial Attack
  5. 2026Robustness of Vision Language Models Against Split-Image Harmful Input Attacks
  6. 2026Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of Refusal
  7. 2023Bergeron: Combating Adversarial Attacks through a Conscience-Based Alignment Framework
  8. 2025On the Impossibility of Separating Intelligence from Judgment: The Computational Intractability of Filtering for AI Alignment