# Interpretable Multi-Modal Hate Speech Detection

arXiv: https://arxiv.org/abs/2103.01616 · HTML: https://mlsanity.com/paper/2103.01616/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 14 of 14)

- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [A Survey Of Methods For Explaining Black Box Models](https://mlsanity.com/paper/1802.01933/) (2018)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Deceiving Google's Perspective API Built for Detecting Toxic Comments](https://mlsanity.com/paper/1702.08138/) (2017)
- [Hate Lingo: A Target-based Linguistic Analysis of Hate Speech in Social Media](https://mlsanity.com/paper/1804.04257/) (2018)
- [All You Need is "Love": Evading Hate-speech Detection](https://mlsanity.com/paper/1808.09115/) (2018)
- [One-step and Two-step Classification for Abusive Language Detection on Twitter](https://mlsanity.com/paper/1706.01206/) (2017)
- [Anyone Can Become a Troll: Causes of Trolling Behavior in Online Discussions](https://mlsanity.com/paper/1702.01119/) (2017)
- [Detecting the Hate Code on Social Media](https://mlsanity.com/paper/1703.05443/) (2017)
- [Cyberbullying Identification Using Participant-Vocabulary Consistency](https://mlsanity.com/paper/1606.08084/) (2016)

## Built on by (most cited first, 10 of 10)

- [A Survey on Multimodal Disinformation Detection](https://mlsanity.com/paper/2103.12541/) (2021)
- [ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings](https://mlsanity.com/paper/2305.14321/) (2023)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Deep Learning for Hate Speech Detection: A Comparative Study](https://mlsanity.com/paper/2202.09517/) (2022)
- [Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate](https://mlsanity.com/paper/2106.05903/) (2021)
- [Rethinking Explainability in the Era of Multimodal AI](https://mlsanity.com/paper/2506.13060/) (2025)
- [Align before Attend: Aligning Visual and Textual Features for Multimodal Hateful Content Detection](https://mlsanity.com/paper/2402.09738/) (2024)
- [Probing Critical Learning Dynamics of PLMs for Hate Speech Detection](https://mlsanity.com/paper/2402.02144/) (2024)
- [Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection](https://mlsanity.com/paper/2412.11745/) (2024)
- [ARC-NLP at Multimodal Hate Speech Event Detection 2023: Multimodal Methods Boosted by Ensemble Learning, Syntactical and Entity Features](https://mlsanity.com/paper/2307.13829/) (2023)

## Abstract (arXiv)

With growing role of social media in shaping public opinions and beliefs across the world, there has been an increased attention to identify and counter the problem of hate speech on social media. Hate speech on online spaces has serious manifestations, including social polarization and hate crimes. While prior works have proposed automated techniques to detect hate speech online, these techniques primarily fail to look beyond the textual content. Moreover, few attempts have been made to focus on the aspects of interpretability of such models given the social and legal implications of incorrect predictions. In this work, we propose a deep neural multi-modal model that can: (a) detect hate speech by effectively capturing the semantics of the text along with socio-cultural context in which a particular hate expression is made, and (b) provide interpretable insights into decisions of our model. By performing a thorough evaluation of different modeling techniques, we demonstrate that our model is able to outperform the existing state-of-the-art hate speech classification approaches. Finally, we show the importance of social and cultural context features towards unearthing clusters associated with different categories of hate.