mlsanityData to 7 Oct 2026

Is GPT-4o mini Blinded by its Own Safety Filters? Exposing the Multimodal-to-Unimodal Bottleneck in Hate Speech Detection

arXiv 2509.13608 ↗First posted 17 Sep 2025cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As Large Multimodal Models (LMMs) become integral to daily digital life, understanding their safety architectures is a critical problem for AI Alignment. This paper presents a systematic analysis of OpenAI's GPT-4o mini, a globally deployed model, on the difficult task of multimodal hate speech detection. Using the Hateful Memes Challenge dataset, we conduct a multi-phase investigation on 500 samples to probe the model's reasoning and failure modes. Our central finding is the experimental identification of a "Unimodal Bottleneck," an architectural flaw where the model's advanced multimodal reasoning is systematically preempted by context-blind safety filters. A quantitative validation of 144 content policy refusals reveals that these overrides are triggered in equal measure by unimodal visual 50% and textual 50% content. We further demonstrate that this safety system is brittle, blocking not only high-risk imagery but also benign, common meme formats, leading to predictable false positives. These findings expose a fundamental tension between capability and safety in state-of-the-art LMMs, highlighting the need for more integrated, context-aware alignment strategies to ensure AI systems can be deployed both safely and effectively.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 20 references that are in the map. First posted on arXiv 17 Sep 2025.

Builds on

10 of 20 references.

References · 10

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  3. 2016Cooperative Inverse Reinforcement Learningcited by 298
  4. 2024Safety of Multimodal Large Language Models on Images and Textscited by 70
  5. 2022Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Featurescited by 55
  6. 2023SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examplescited by 53
  7. 2024textttModSCAN: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalitiescited by 21
  8. 2025Demystifying Hateful Content: Leveraging Large Multimodal Models for Hateful Meme Detection with Explainable Decisionscited by 13
  9. 2025From Recognition to Reasoning: Advancing Multimodal Harmful Meme Detection via Chain-of-Thought Alignmentcited by 3
  10. 2025Detecting and Understanding Hateful Contents in Memes Through Captioning and Visual Question-Answeringcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Is AI Ready for Multimodal Hate Speech Detection? A Comprehensive Dataset and Benchmark Evaluation ↗
  2. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memes
  3. 2025Robust Adaptation of Large Multimodal Models for Retrieval Augmented Hateful Meme Detection
  4. 2023Detecting and Correcting Hate Speech in Multimodal Memes with Large Visual Language Model
  5. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memes
  6. 2025Reasoning-Aware Multimodal Fusion for Hateful Video Detection
  7. 2026FBHM: Functional Benchmarking and Steering of VLMs for Hateful Meme Detection
  8. 2025Labels or Input? Rethinking Augmentation in Multimodal Hate Detection