mlsanityData to 7 Oct 2026

Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating

arXiv 2603.12598 ↗First posted 13 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have shown remarkable potential across a wide array of vision-language tasks, leading to their adoption in critical domains such as finance and healthcare. However, their growing deployment also introduces significant security and privacy risks. Malicious actors could potentially exploit these models to extract sensitive information, highlighting a critical vulnerability. Recent studies show that LVLMs often fail to consistently refuse instructions designed to compromise user privacy. While existing work on privacy protection has made meaningful progress in preventing the leakage of sensitive data, they are constrained by limitations in both generalization and non-destructiveness. They often struggle to robustly handle unseen privacy-related queries and may inadvertently degrade a model's performance on standard tasks. To address these challenges, we introduce Neural Gate, a novel method for mitigating privacy risks through neuron-level model editing. Our method improves a model's privacy safeguards by increasing its rate of refusal for privacy-related questions, crucially extending this protective behavior to novel sensitive queries not encountered during the editing process. Neural Gate operates by learning a feature vector to identify neurons associated with privacy-related concepts within the model's representation of a subject. This localization then precisely guides the update of model parameters. Through comprehensive experiments on MiniGPT and LLaVA, we demonstrate that our method significantly boosts the model's privacy protection while preserving its original utility. The code is available at https://github.com/Xiangkui-Cao/Neural-Gate.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 43 references that are in the map. First posted on arXiv 13 Mar 2026.

Builds on

30 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models11 Jun
  2. 2024Private Attribute Inference from Images with Vision-Language Models16 Apr

Other references · 28

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2022Mass-Editing Memory in a Transformercited by 976
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
16 more
  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 224
  3. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  4. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  5. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  6. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  7. 2017Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Imagescited by 101
  8. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  9. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  10. 2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 65
  11. 2022Privately Fine-Tuning Large Language Models with Differential Privacycited by 39
  12. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  13. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 30
  14. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Modelscited by 25
  15. 2024Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMscited by 6
  16. 2025REVAL: A Comprehension Evaluation on Reliability and Values of Large Vision-Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?
  2. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden States
  3. 2025VIP: Visual Information Protection through Adversarial Attacks on Vision-Language Models
  4. 2025Rethinking Post-Unlearning Behavior of Large Vision-Language Models
  5. 2023Can Language Models be Instructed to Protect Personal Information?
  6. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models
  7. 2024A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends
  8. 2024Privacy-Aware Visual Language Models