# Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media

arXiv: https://arxiv.org/abs/2307.09312 · HTML: https://mlsanity.com/paper/2307.09312/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 18 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [Large Scale Crowdsourcing and Characterization of Twitter Abusive Behavior](https://mlsanity.com/paper/1802.00393/) (2018)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [HateBERT: Retraining BERT for Abusive Language Detection in English](https://mlsanity.com/paper/2010.12472/) (2020)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Media](https://mlsanity.com/paper/1910.12574/) (2019)
- [Prompting for Multimodal Hateful Meme Classification](https://mlsanity.com/paper/2302.04156/) (2023)
- [Quantifying social organization and political polarization in online platforms](https://mlsanity.com/paper/2010.00590/) (2020)
- [Understanding Political Polarization via Jointly Modeling Users, Connections and Multimodal Contents on Heterogeneous Graphs](https://mlsanity.com/paper/2201.05946/) (2022)
- [Predicting Hateful Discussions on Reddit using Graph Transformer Networks and Communal Context](https://mlsanity.com/paper/2301.04248/) (2023)
- [Abusive Language Detection in Heterogeneous Contexts: Dataset Collection and the Role of Supervised Attention](https://mlsanity.com/paper/2105.11119/) (2021)
- [TweetNERD -- End to End Entity Linking Benchmark for Tweets](https://mlsanity.com/paper/2210.08129/) (2022)
- [Towards A Multi-agent System for Online Hate Speech Detection](https://mlsanity.com/paper/2105.01129/) (2021)
- [Qualitative Analysis of a Graph Transformer Approach to Addressing Hate Speech: Adapting to Dynamically Changing Content](https://mlsanity.com/paper/2301.10871/) (2023)
- [Robust Candidate Generation for Entity Linking on Short Social Media Texts](https://mlsanity.com/paper/2210.07472/) (2022)

## Built on by (most cited first, 6 of 6)

- [A Survey of Graph Transformers: Architectures, Theories and Applications](https://mlsanity.com/paper/2502.16533/) (2025)
- [Trustworthy Hate Speech Detection Through Visual Augmentation](https://mlsanity.com/paper/2409.13557/) (2024)
- [Graphically Speaking: Unmasking Abuse in Social Media with Conversation Insights](https://mlsanity.com/paper/2504.01902/) (2025)
- [MEMEWEAVER: Inter-Meme Graph Reasoning for Sexism and Misogyny Detection](https://mlsanity.com/paper/2601.08684/) (2026)
- [Co-AttenDWG: Co-Attentive Dimension-Wise Gating and Expert Fusion for Multi-Modal Offensive Content Detection](https://mlsanity.com/paper/2505.19010/) (2025)
- [Community Norms in the Spotlight: Enabling Task-Agnostic Unsupervised Pre-Training to Benefit Online Social Media](https://mlsanity.com/paper/2602.02525/) (2026)

## Abstract (arXiv)

We present the Multi-Modal Discussion Transformer (mDT), a novel methodfor detecting hate speech in online social networks such as Reddit discussions. In contrast to traditional comment-only methods, our approach to labelling a comment as hate speech involves a holistic analysis of text and images grounded in the discussion context. This is done by leveraging graph transformers to capture the contextual relationships in the discussion surrounding a comment and grounding the interwoven fusion layers that combine text and image embeddings instead of processing modalities separately. To evaluate our work, we present a new dataset, HatefulDiscussions, comprising complete multi-modal discussions from multiple online communities on Reddit. We compare the performance of our model to baselines that only process individual comments and conduct extensive ablation studies.