Is GPT-4o mini Blinded by its Own Safety Filters? Exposing the Multimodal-to-Unimodal Bottleneck in Hate Speech Detection
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
As Large Multimodal Models (LMMs) become integral to daily digital life, understanding their safety architectures is a critical problem for AI Alignment. This paper presents a systematic analysis of OpenAI's GPT-4o mini, a globally deployed model, on the difficult task of multimodal hate speech detection. Using the Hateful Memes Challenge dataset, we conduct a multi-phase investigation on 500 samples to probe the model's reasoning and failure modes. Our central finding is the experimental identification of a "Unimodal Bottleneck," an architectural flaw where the model's advanced multimodal reasoning is systematically preempted by context-blind safety filters. A quantitative validation of 144 content policy refusals reveals that these overrides are triggered in equal measure by unimodal visual 50% and textual 50% content. We further demonstrate that this safety system is brittle, blocking not only high-risk imagery but also benign, common meme formats, leading to predictable false positives. These findings expose a fundamental tension between capability and safety in state-of-the-art LMMs, highlighting the need for more integrated, context-aware alignment strategies to ensure AI systems can be deployed both safely and effectively.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 20 references that are in the map. First posted on arXiv 17 Sep 2025.
Builds on
10 of 20 references.
References · 10
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
- 2016Cooperative Inverse Reinforcement Learningcited by 298
- 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
- 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
- 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
- 2024textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalitiescited by 21
- 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 13
- 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
- 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answeringcited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Is AI Ready for Multimodal Hate Speech Detection? A Comprehensive Dataset and Benchmark Evaluation ↗
- 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes
- 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection
- 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
- 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes
- 2025Reasoning-Aware Multimodal Fusion for Hateful Video Detection
- 2026FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection
- 2025Labels or Input? Rethinking Augmentation in Multimodal Hate Detection