mlsanityData to 7 Oct 2026

Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations

arXiv 2406.11801 ↗First posted 17 Jun 2024cs.CLcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Ensuring the safe alignment of large language models (LLMs) with human values is critical as they become integral to applications like translation and question answering. Current alignment methods struggle with dynamic user intentions and complex objectives, making models vulnerable to generating harmful content. We propose Safety Arithmetic, a training-free framework enhancing LLM safety across different scenarios: Base models, Supervised fine-tuned models (SFT), and Edited models. Safety Arithmetic involves Harm Direction Removal to avoid harmful content and Safety Alignment to promote safe responses. Additionally, we present NoIntentEdit, a dataset highlighting edit instances that could compromise model safety if used unintentionally. Our experiments show that Safety Arithmetic significantly improves safety measures, reduces over-safety, and maintains model utility, outperforming existing methods in ensuring safe content generation.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 54 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

40 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching22 May
  2. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
25 more
  1. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  3. 2022Mass-Editing Memory in a Transformercited by 976
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2022Transformers learn in-context by gradient descentcited by 714
  8. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2023Larger language models do in-context learning differentlycited by 386
  12. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  13. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  14. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  15. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  16. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  17. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  18. 2022Patching open-vocabulary models by interpolating weightscited by 187
  19. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  20. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  21. 2023On the Exploitability of Instruction Tuningcited by 110
  22. 2022On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Modelcited by 78
  23. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
  24. 2023OpinionGPT: Modelling Explicit Biases in Instruction-Tuned LLMscited by 29
  25. 2024Sowing the Wind, Reaping the Whirlwind: The Impact of Editing Language Modelscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 10.

32 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  3. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 23
  4. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19
  5. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 15
  6. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  7. 2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
  8. 2025Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignmentcited by 10
  9. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  10. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 9
  11. 2026Why Steering Works: Toward a Unified View of Language Model Parameter Dynamicscited by 8
  12. 2025Attributional Safety Failures in Large Language Models under Code-Mixed Perturbationscited by 6
20 more
  1. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  2. 2026Fine-Grained Activation Steering: Steering Less, Achieving Morecited by 6
  3. 2025Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledgecited by 5
  4. 2025Chain-of-Lure: A Universal Jailbreak Attack Framework using Unconstrained Synthetic Narrativescited by 5
  5. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  6. 2025Large Language Model Sourcing: A Surveycited by 5
  7. 2025BADTV: Unveiling Backdoor Threats in Third-Party Task Vectorscited by 3
  8. 2025MemeSense: An Adaptive In-Context Framework for Social Commonsense Driven Meme Moderationcited by 3
  9. 2026SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systemscited by 3
  10. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  11. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  12. 2025Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questionscited by 1
  13. 2025Safety Alignment Depth in Large Language Models: A Markov Chain Perspectivecited by 1
  14. 2025Reimagining Safety Alignment with An Imagecited by 1
  15. 2025ProSocialAlign: Preference Conditioned Test Time Alignment in Language Modelscited by 1
  16. 2026Tutoring Large Language Models to be Domain-adaptive, Precise, and Safecited by 0
  17. 2026SafeMath: Safe Solutions for Unsafe Math Word Problemscited by 0
  18. 2026Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Modelscited by 0
  19. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  20. 2026Tutoring Large Language Models to be Domain-adaptive, Precise and Safecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  2. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion
  3. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements
  4. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  5. 2024SafeInfer: Context Adaptive Decoding Time Safety Alignment for Large Language Models
  6. 2026Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
  7. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  8. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization