# Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Models

arXiv: https://arxiv.org/abs/2503.20807 · HTML: https://mlsanity.com/paper/2503.20807/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 12 of 12)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [OctoPack: Instruction Tuning Code Large Language Models](https://mlsanity.com/paper/2308.07124/) (2023)
- [Platypus: Quick, Cheap, and Powerful Refinement of LLMs](https://mlsanity.com/paper/2308.07317/) (2023)
- [Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models](https://mlsanity.com/paper/2405.16833/) (2024)
- [Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey](https://mlsanity.com/paper/2409.18169/) (2024)
- [SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection](https://mlsanity.com/paper/2410.07471/) (2024)
- [Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models](https://mlsanity.com/paper/2405.17374/) (2024)

## Built on by (most cited first, 17 of 17)

- [Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets](https://mlsanity.com/paper/2506.05346/) (2025)
- [The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems](https://mlsanity.com/paper/2507.02097/) (2025)
- [Unveiling the Basin-Like Loss Landscape in Large Language Models](https://mlsanity.com/paper/2505.17646/) (2025)
- [Shape it Up! Restoring LLM Safety during Finetuning](https://mlsanity.com/paper/2505.17196/) (2025)
- [When Thinking Backfires: Mechanistic Insights Into Reasoning-Induced Misalignment](https://mlsanity.com/paper/2509.00544/) (2025)
- [Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influence](https://mlsanity.com/paper/2603.00498/) (2026)
- [Discovering Hierarchical Latent Capabilities of Language Models via Causal Representation Learning](https://mlsanity.com/paper/2506.10378/) (2025)
- [SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization](https://mlsanity.com/paper/2511.06222/) (2025)
- [Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models](https://mlsanity.com/paper/2602.02136/) (2026)
- [ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Concepts](https://mlsanity.com/paper/2508.16325/) (2025)
- [Uncovering Intrinsic Capabilities: A Paradigm for Data Curation in Vision-Language Models](https://mlsanity.com/paper/2510.00040/) (2025)
- [GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning](https://mlsanity.com/paper/2603.10243/) (2026)
- [Alignment Dynamics in LLM Fine-Tuning](https://mlsanity.com/paper/2605.18309/) (2026)
- [Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning](https://mlsanity.com/paper/2602.17546/) (2026)
- [From Parameter Dynamics to Risk Scoring : Quantifying Sample-Level Safety Degradation in LLM Fine-tuning](https://mlsanity.com/paper/2605.04572/) (2026)
- [The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems](https://mlsanity.com/paper/2605.23024/) (2026)
- [Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training](https://mlsanity.com/paper/2606.26102/) (2026)

## Abstract (arXiv)

Fine-tuning Large Language Models (LLMs) on some task-specific datasets has been a primary use of LLMs. However, it has been empirically observed that this approach to enhancing capability inevitably compromises safety, a phenomenon also known as the safety-capability trade-off in LLM fine-tuning. This paper presents a theoretical framework for understanding the interplay between safety and capability in two primary safety-aware LLM fine-tuning strategies, providing new insights into the effects of data similarity, context overlap, and alignment loss landscape. Our theoretical results characterize the fundamental limits of the safety-capability trade-off in LLM fine-tuning, which are also validated by numerical experiments.