mlsanityData to 14 Oct 2026

Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models

arXiv 2405.17374 ↗First posted 27 May 2024cs.LGcited by 46

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment is crucial to ensure that large language models (LLMs) behave in ways that align with human preferences and prevent harmful actions during inference. However, recent studies show that the alignment can be easily compromised through finetuning with only a few adversarially designed training examples. We aim to measure the risks in finetuning LLMs through navigating the LLM safety landscape. We discover a new phenomenon observed universally in the model parameter space of popular open-source LLMs, termed as "safety basin": random perturbations to model weights maintain the safety level of the original aligned model within its local neighborhood. However, outside this local region, safety is fully compromised, exhibiting a sharp, step-like drop. This safety basin contrasts sharply with the LLM capability landscape, where model performance peaks at the origin and gradually declines as random perturbation increases. Our discovery inspires us to propose the new VISAGE safety metric that measures the safety in LLM finetuning by probing its safety landscape. Visualizing the safety landscape of the aligned model enables us to understand how finetuning compromises safety by dragging the model away from the safety basin. The LLM safety landscape also highlights the system prompt's critical role in protecting a model, and that such protection transfers to its perturbed variants within the safety basin. These observations from our safety landscape research provide new insights for future work on LLM safety community. Our code is publicly available at https://github.com/ShengYun-Peng/llm-landscape.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 53 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

43 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  2. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Are aligned neural networks adversarially aligned?26 Jun
  7. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  8. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 35

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2023Mistral 7Bcited by 3,164
23 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  7. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  11. 2017Sharp Minima Can Generalize For Deep Netscited by 466
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  13. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  14. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  15. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  16. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 191
  17. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  18. 2020Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustnesscited by 132
  19. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  20. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 86
  21. 2023Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4cited by 48
  22. 2016PhaseMax: Convex Phase Retrieval via Basis Pursuitcited by 25
  23. 2020Discriminative Adversarial Search for Abstractive Summarizationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 24; 2026: 10.

46 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  3. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  4. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  5. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  6. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  7. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  8. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 37
  9. 2024Locking Down the Finetuned LLMs Safetycited by 22
  10. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  11. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 20
  12. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
34 more
  1. 2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 17
  2. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  3. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  4. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12
  5. 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
  6. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  7. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  8. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  9. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  10. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
  11. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  12. 2025Bridging Symmetry and Robustness: On the Role of Equivariance in Enhancing Adversarial Robustnesscited by 8
  13. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  14. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  15. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
  16. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  17. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  18. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  19. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  20. 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuningcited by 5
  21. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  22. 2025A Framework for Situating Innovations, Opportunities, and Challenges in Advancing Vertical Systems with Large AI Modelscited by 2
  23. 2025SafetyPairs: Isolating Safety Critical Image Features with Counterfactual Image Generationcited by 2
  24. 2026Layered LLM Defenses as an Ensemble: Access Tiers, Inference Cost, and the Measured Failure Correlation Between Defense Layerscited by 2
  25. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  26. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Datacited by 0
  27. 2025BIRD: Behavior Induction via Representation-structure Distillationcited by 0
  28. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  29. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learningcited by 0
  30. 2026A Gravitational Interpretation of Safety Reversion under Fine-Tuningcited by 0
  31. 2026The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persistscited by 0
  32. 2026Beyond Quadratic Loss: The Stability Phase Diagram of Adamcited by 0
  33. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  34. 2026How Does Local Landscape Geometry Evolve in Language Model Pre-Training?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
  2. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  3. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  4. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
  5. 2026SafeAnchor: Preventing Cumulative Safety Erosion in Continual Domain Adaptation of Large Language Models
  6. 2026Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
  7. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  8. 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patching