Data to 7 Oct 2026

Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack

arXiv 2405.18641 ↗First posted 28 May 2024cs.LGcited by 89

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more

LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Recent studies show that Large Language Models (LLMs) with safety alignment can be jail-broken by fine-tuning on a dataset mixed with harmful data. First time in the literature, we show that the jail-broken effect can be mitigated by separating states in the finetuning stage to optimize the alignment and user datasets. Unfortunately, our subsequent study shows that this simple Bi-State Optimization (BSO) solution experiences convergence instability when steps invested in its alignment state is too small, leading to downgraded alignment performance. By statistical analysis, we show that the textitexcess drift towards consensus could be a probable reason for the instability. To remedy this issue, we propose Lazy(i) safety alignment (Lisa), which introduces a proximal term to constraint the drift of each state. Theoretically, the benefit of the proximal term is supported by the convergence analysis, wherein we show that a sufficient large proximal factor is necessary to guarantee Lisa's convergence. Empirically, our results on four downstream finetuning tasks show that Lisa with a proximal term can significantly increase alignment performance while maintaining the LLM's accuracy on the user tasks. Code is available at urlhttps://github.com/git-disl/Lisa.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 91 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

54 of 91 references.

Closest ancestors · 13

Same direction, newest first

1 more

2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 41

2022Training language models to follow instructions with human feedbackcited by 15,712
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2016Overcoming catastrophic forgetting in neural networkscited by 4,718
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
2023Mistral 7Bcited by 3,065
2018Federated Optimization in Heterogeneous Networkscited by 2,971
2015Character-level Convolutional Networks for Text Classificationcited by 2,293
2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326

29 more

2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2021Federated Learning Based on Dynamic Regularizationcited by 555
2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
2019Meta-Learning with Implicit Gradientscited by 523
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
2023Preference Ranking Optimization for Human Alignmentcited by 283
2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
2024A Survey on Large Language Model-Based Game Agentscited by 106
2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
2020Defending against Backdoors in Federated Learning with Robust Learning Ratecited by 82
2021FedCM: Federated Learning with Client-level Momentumcited by 77
2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
2023Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectivescited by 48
2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
2024Immunization against harmful fine-tuning attackscited by 33
2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24
2023BERT4ETH: A Pre-trained Transformer for Ethereum Fraud Detectioncited by 19
2021Proximal Gradient Descent-Ascent: Variable Convergence under KL Geometrycited by 17
2014Douglas-Rachford splitting for nonconvex optimization with application to nonconvex feasibility problemscited by 15
2023Fusion of Global and Local Knowledge for Personalized Federated Learningcited by 14
2019Progressive DNN Compression: A Key to Achieve Ultra-High Weight Pruning and Quantization Rates using ADMMcited by 3
2018An Algorithmic Framework of Variable Metric Over-Relaxed Hybrid Proximal Extra-Gradient Methodcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 39; 2026: 33.

89 papers.

Sort:

By area:

2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27

38 more

2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
2024Locking Down the Finetuned LLMs Safetycited by 22
2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19
2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
2024Superficial Safety Alignment Hypothesiscited by 14
2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
2024Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Mergingcited by 11
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
2025Self-Destructive Language Modelcited by 10
2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8
2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 7
2026Understanding and Preserving Safety in Fine-Tuned LLMscited by 7
2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
2025Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearningcited by 5
2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
2024Evaluating Defences against Unsafe Feedback in RLHFcited by 4
2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4
2025Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Modelscited by 4
2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 4

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way