# SGM: Safety Glasses for Multimodal Large Language Models via Neuron-Level Detoxification

arXiv: https://arxiv.org/abs/2512.15052 · HTML: https://mlsanity.com/paper/2512.15052/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding](https://mlsanity.com/paper/2402.08983/) (2024)
- [MLLM-Protector: Ensuring MLLM's Safety without Hurting Performance](https://mlsanity.com/paper/2401.02906/) (2024)
- [Safety Assessment of Chinese Large Language Models](https://mlsanity.com/paper/2304.10436/) (2023)
- [Whispering Experts: Neural Interventions for Toxicity Mitigation in Language Models](https://mlsanity.com/paper/2407.12824/) (2024)
- [CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration](https://mlsanity.com/paper/2409.11365/) (2024)
- [BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against Jailbreak Attacks](https://mlsanity.com/paper/2410.20971/) (2024)
- [Safeguarding Vision-Language Models Against Patched Visual Prompt Injectors](https://mlsanity.com/paper/2405.10529/) (2024)
- [Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models](https://mlsanity.com/paper/2407.21659/) (2024)
- [Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models](https://mlsanity.com/paper/2410.09047/) (2024)
- [Fairness and Bias in Multimodal AI: A Survey](https://mlsanity.com/paper/2406.19097/) (2024)
- [Self-conditioning pre-trained language models](https://mlsanity.com/paper/2110.02802/) (2021)
- [Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models](https://mlsanity.com/paper/2511.06793/) (2025)
- [Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs](https://mlsanity.com/paper/2503.00037/) (2025)

## Built on by (most cited first, 1 of 1)

- [TraceRouter: Robust Safety for Large Foundation Models via Path-Level Intervention](https://mlsanity.com/paper/2601.21900/) (2026)

## Abstract (arXiv)

Disclaimer: Samples in this paper may be harmful and cause discomfort. Multimodal large language models (MLLMs) enable multimodal understanding but inherit toxic signals from weakly curated pretraining corpora, leading to explicitly toxic outputs, especially under adversarial triggers that late, opaque training-free detoxification methods struggle to handle. We propose SGM, a white-box neuron-level multimodal intervention that acts like safety glasses for toxic neurons: it recalibrates a set of toxicity-associated neurons via expertise-weighted soft suppression, neutralizing harmful cross-modal activations without any parameter updates. We establish MM-TOXIC-QA, a multimodal toxicity data framework, and compare SGM with existing detoxification techniques. Experiments on open-source MLLMs show that SGM mitigates explicit toxicity in standard and adversarial conditions, cutting average harmful rates from 45.0% to 4.5% while preserving fluency and multimodal reasoning. SGM is extensible, and its combined defenses, denoted as SGM*, integrate with existing detoxification methods for stronger safety performance, providing an interpretable, low-cost solution for toxicity-controlled multimodal generation.