Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
Safety alignment in Large Language Models (LLMs) inherently presents a multi-objective optimization conflict, often accompanied by an unintended degradation of general capabilities. Existing mitigation strategies typically rely on global gradient geometry to resolve these conflicts, yet they overlook Modular Heterogeneity within Transformers, specifically that the functional sensitivity and degree of conflict vary substantially across different attention heads. Such global approaches impose uniform update rules across all parameters, often resulting in suboptimal trade-offs by indiscriminately updating utility sensitive heads that exhibit intense gradient conflicts. To address this limitation, we propose Conflict-Aware Sparse Tuning (CAST), a framework that integrates head-level diagnosis with sparse fine-tuning. CAST first constructs a pre-alignment conflict map by synthesizing Optimization Conflict and Functional Sensitivity, which then guides the selective update of parameters. Experiments reveal that alignment conflicts in LLMs are not uniformly distributed. We find that the drop in general capabilities mainly comes from updating a small group of ``high-conflict'' heads. By simply skipping these heads during training, we significantly reduce this loss without compromising safety, offering an interpretable and parameter-efficient approach to improving the safety-utility trade-off.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 24 references that are in the map. First posted on arXiv 7 Jan 2026.
Builds on
20 of 24 references.
References · 20
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2020Measuring Massive Multitask Language Understandingcited by 7,768
- 2023Let's Verify Step by Stepcited by 3,598
- 2024Qwen2 Technical Reportcited by 2,076
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2020Gradient Surgery for Multi-Task Learningcited by 1,210
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
8 more
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
- 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
- 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
- 2023Mitigating the Alignment Tax of RLHFcited by 181
- 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models
- 2025Gradient Surgery for Safe LLM Fine-Tuning
- 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
- 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
- 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization
- 2025Black-Box Behavioral Distillation Breaks Safety Alignment in Medical LLMs
- 2025Curvature-Aware Safety Restoration In LLMs Fine-Tuning
- 2026Understanding and Preserving Safety in Fine-Tuned LLMs