SGM: Safety Glasses for Multimodal Large Language Models via Neuron-Level Detoxification
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
Abstract (arXiv)
Disclaimer: Samples in this paper may be harmful and cause discomfort. Multimodal large language models (MLLMs) enable multimodal understanding but inherit toxic signals from weakly curated pretraining corpora, leading to explicitly toxic outputs, especially under adversarial triggers that late, opaque training-free detoxification methods struggle to handle. We propose SGM, a white-box neuron-level multimodal intervention that acts like safety glasses for toxic neurons: it recalibrates a set of toxicity-associated neurons via expertise-weighted soft suppression, neutralizing harmful cross-modal activations without any parameter updates. We establish MM-TOXIC-QA, a multimodal toxicity data framework, and compare SGM with existing detoxification techniques. Experiments on open-source MLLMs show that SGM mitigates explicit toxicity in standard and adversarial conditions, cutting average harmful rates from 45.0% to 4.5% while preserving fluency and multimodal reasoning. SGM is extensible, and its combined defenses, denoted as SGM*, integrate with existing detoxification methods for stronger safety performance, providing an interpretable, low-cost solution for toxicity-controlled multimodal generation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 41 references that are in the map. First posted on arXiv 17 Dec 2025.
Builds on
16 of 41 references.
Closest ancestors · 8
Same direction, newest first
- 2025Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models10 Nov
- 2025Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs25 Feb
- 2024BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks28 Oct
- 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models11 Oct
- 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration17 Sep
- 2024Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models31 Jul
- 2024Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors17 May
- 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance5 Jan
Other references · 8
- 2023GPT-4 Technical Reportcited by 18,132
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
- 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
- 2023Safety Assessment of Chinese Large Language Modelscited by 82
- 2024Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Modelscited by 31
- 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
- 2021Self-conditioning pre-trained language modelscited by 16
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models
- 2024Detoxifying Large Language Models via Knowledge Editing
- 2025Rethinking Toxicity Evaluation in Large Language Models: A Multi-Label Perspective
- 2024Large Language Models can be Strong Self-Detoxifiers
- 2025Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
- 2026Detoxifying LLMs via Representation Erasure-Based Preference Optimization
- 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
- 2025Circumventing Safety Alignment in Large Language Models Through Embedding Space Toxicity Attenuation