# MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework

arXiv: https://arxiv.org/abs/2409.05136 · HTML: https://mlsanity.com/paper/2409.05136/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 52 references that are in the map.
- First posted on arXiv 8 Sep 2024.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [ETHOS: an Online Hate Speech Detection Dataset](https://mlsanity.com/paper/2006.08328/) (2020)
- [Disentangling Hate in Online Memes](https://mlsanity.com/paper/2108.06207/) (2021)
- [Detecting Hate Speech in Multi-modal Memes](https://mlsanity.com/paper/2012.14891/) (2020)
- ["Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning](https://mlsanity.com/paper/2104.05947/) (2021)
- [Emotion Based Hate Speech Detection using Multimodal Learning](https://mlsanity.com/paper/2202.06218/) (2022)
- [Character-level HyperNetworks for Hate Speech Detection](https://mlsanity.com/paper/2111.06336/) (2021)
- [Selecting and combining complementary feature representations and classifiers for hate speech detection](https://mlsanity.com/paper/2201.06721/) (2022)
- [QUARC: Quaternion Multi-Modal Fusion Architecture For Hate Speech Classification](https://mlsanity.com/paper/2012.08312/) (2020)
- [Towards A Multi-agent System for Online Hate Speech Detection](https://mlsanity.com/paper/2105.01129/) (2021)
- [Overview of Memotion 3: Sentiment and Emotion Analysis of Codemixed Hinglish Memes](https://mlsanity.com/paper/2309.06517/) (2023)

## Built on by (most cited first, 2 of 2)

- [Multi-task Learning with Active Learning for Arabic Offensive Speech Detection](https://mlsanity.com/paper/2506.02753/) (2025)
- [Towards Training-free Multimodal Hate Localisation with Large Language Models](https://mlsanity.com/paper/2602.09637/) (2026)

## Abstract (arXiv)

Social media has a significant impact on people's lives. Hate speech on social media has emerged as one of society's most serious issues in recent years. Text and pictures are two forms of multimodal data that are distributed within articles. Unimodal analysis has been the primary emphasis of earlier approaches. Additionally, when doing multimodal analysis, researchers neglect to preserve the distinctive qualities associated with each modality. To address these shortcomings, the present article suggests a scalable architecture for multimodal hate content detection called transformer-based multilevel attention (STMA). This architecture consists of three main parts: a combined attention-based deep learning mechanism, a vision attention-mechanism encoder, and a caption attention-mechanism encoder. To identify hate content, each component uses various attention processes and handles multimodal data in a unique way. Several studies employing multiple assessment criteria on three hate speech datasets such as Hateful memes, MultiOff, and MMHS150K, validate the suggested architecture's efficacy. The outcomes demonstrate that on all three datasets, the suggested strategy performs better than the baseline approaches.