mlsanityData to 14 Oct 2026

Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models

arXiv 2505.16104 ↗First posted 22 May 2025cs.CL cs.CV cs.LGcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

With the increasing size of Large Vision-Language Models (LVLMs), network pruning techniques aimed at compressing models for deployment in resource-constrained environments have garnered significant attention. However, we observe that pruning often leads to a degradation in safety performance. To address this issue, we present a novel and lightweight approach, termed Hierarchical Safety Realignment (HSR). HSR operates by first quantifying the contribution of each attention head to safety, identifying the most critical ones, and then selectively restoring neurons directly within these attention heads that play a pivotal role in maintaining safety. This process hierarchically realigns the safety of pruned LVLMs, progressing from the attention head level to the neuron level. We validate HSR across various models and pruning strategies, consistently achieving notable improvements in safety performance. To our knowledge, this is the first work explicitly focused on restoring safety in LVLMs post-pruning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 31 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

22 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations15 Nov
  2. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt6 Jun
  3. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting14 Mar
  4. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov

Other references · 18

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  11. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  12. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
6 more
  1. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  2. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 66
  3. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 54
  4. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  5. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  6. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuningcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Resultscited by 2
  2. 2026AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Modelscited by 1
  3. 2026EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reusecited by 1
  4. 2026Safety-Potential Pruning for Enhancing Safety Prompts Against VLM Jailbreaking Without Retrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Targeted Interpretable Safety Neuron Enhancement for Multilingual Vision-Language Large Models
  2. 2026Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection
  3. 2026HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Models
  4. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Models
  5. 2026One Anchor for All: Unified Multilingual and Multimodal Safety Alignment for LVLMs
  6. 2026Seeing No Evil: Blinding Large Vision-Language Models to Safety Instructions via Adversarial Attention Hijacking
  7. 2024A safety realignment framework via subspace-oriented model fusion for large language models
  8. 2026Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models