# "Subverting the Jewtocracy": Online Antisemitism Detection Using Multimodal Deep Learning

arXiv: https://arxiv.org/abs/2104.05947 · HTML: https://mlsanity.com/paper/2104.05947/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 13 Apr 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Deep Learning for Hate Speech Detection in Tweets](https://mlsanity.com/paper/1706.00188/) (2017)
- [A Benchmark Dataset for Learning to Intervene in Online Hate Speech](https://mlsanity.com/paper/1909.04251/) (2019)
- [Exploring Hate Speech Detection in Multimodal Publications](https://mlsanity.com/paper/1910.03814/) (2019)
- [MFAS: Multimodal Fusion Architecture Search](https://mlsanity.com/paper/1903.06496/) (2019)
- [One-step and Two-step Classification for Abusive Language Detection on Twitter](https://mlsanity.com/paper/1706.01206/) (2017)
- [A Unified Deep Learning Architecture for Abuse Detection](https://mlsanity.com/paper/1802.00385/) (2018)
- [Mean Birds: Detecting Aggression and Bullying on Twitter](https://mlsanity.com/paper/1702.06877/) (2017)
- [Hate Speech in Pixels: Detection of Offensive Memes towards Automatic Moderation](https://mlsanity.com/paper/1910.02334/) (2019)
- [A Quantitative Approach to Understanding Online Antisemitism](https://mlsanity.com/paper/1809.01644/) (2018)
- [What is Gab? A Bastion of Free Speech or an Alt-Right Echo Chamber?](https://mlsanity.com/paper/1802.05287/) (2018)
- [Multi-label Categorization of Accounts of Sexism using a Neural Framework](https://mlsanity.com/paper/1910.04602/) (2019)
- [Annotating Antisemitic Online Content. Towards an Applicable Definition of Antisemitism](https://mlsanity.com/paper/1910.01214/) (2019)
- [AbuseAnalyzer: Abuse Detection, Severity and Target Prediction for Gab Posts](https://mlsanity.com/paper/2010.00038/) (2020)

## Built on by (most cited first, 21 of 21)

- [MOMENTA: A Multimodal Framework for Detecting Harmful Memes and Their Targets](https://mlsanity.com/paper/2109.05184/) (2021)
- [Detecting and Understanding Harmful Memes: A Survey](https://mlsanity.com/paper/2205.04274/) (2022)
- [Detecting Propaganda Techniques in Memes](https://mlsanity.com/paper/2109.08013/) (2021)
- [Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots](https://mlsanity.com/paper/2209.03463/) (2022)
- [What do you MEME? Generating Explanations for Visual Semantic Role Labelling in Memes](https://mlsanity.com/paper/2212.00715/) (2022)
- [BanglaAbuseMeme: A Dataset for Bengali Abusive Meme Classification](https://mlsanity.com/paper/2310.11748/) (2023)
- [MEMEX: Detecting Explanatory Evidence for Memes via Knowledge-Enriched Contextualization](https://mlsanity.com/paper/2305.15913/) (2023)
- [Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations](https://mlsanity.com/paper/2402.12198/) (2024)
- [Codes, Patterns and Shapes of Contemporary Online Antisemitism and Conspiracy Narratives -- an Annotation Guide and Labeled German-Language Dataset in the Context of COVID-19](https://mlsanity.com/paper/2210.07934/) (2022)
- [Antisemitic Messages? A Guide to High-Quality Annotation and a Labeled Dataset of Tweets](https://mlsanity.com/paper/2304.14599/) (2023)
- [Factoring Hate Speech: A New Annotation Framework to Study Hate Speech in Social Media](https://mlsanity.com/paper/2311.03969/) (2023)
- [Using LLMs to discover emerging coded antisemitic hate-speech in extremist social media](https://mlsanity.com/paper/2401.10841/) (2024)
- [Efficient Hate Speech Detection: Evaluating 38 Models from Traditional Methods to Transformers](https://mlsanity.com/paper/2509.14266/) (2025)
- [hate-alert@DravidianLangTech-ACL2022: Ensembling Multi-Modalities for Tamil TrollMeme Classification](https://mlsanity.com/paper/2204.12587/) (2022)
- [Detection of Propaganda Techniques in Visuo-Lingual Metaphor in Memes](https://mlsanity.com/paper/2205.02937/) (2022)
- [Detecting the Role of an Entity in Harmful Memes: Techniques and Their Limitations](https://mlsanity.com/paper/2205.04402/) (2022)
- [MHS-STMA: Multimodal Hate Speech Detection via Scalable Transformer-Based Multilevel Attention Framework](https://mlsanity.com/paper/2409.05136/) (2024)
- [TeamX@DravidianLangTech-ACL2022: A Comparative Analysis for Troll-Based Meme Classification](https://mlsanity.com/paper/2205.04404/) (2022)
- [Monitoring the evolution of antisemitic discourse on extremist social media using BERT](https://mlsanity.com/paper/2403.05548/) (2024)
- [A Group-Specific Approach to NLP for Hate Speech Detection](https://mlsanity.com/paper/2304.11223/) (2023)
- [Evaluating Large Language Models for Antisemitic Incident Classification](https://mlsanity.com/paper/2607.04890/) (2026)

## Abstract (arXiv)

The exponential rise of online social media has enabled the creation, distribution, and consumption of information at an unprecedented rate. However, it has also led to the burgeoning of various forms of online abuse. Increasing cases of online antisemitism have become one of the major concerns because of its socio-political consequences. Unlike other major forms of online abuse like racism, sexism, etc., online antisemitism has not been studied much from a machine learning perspective. To the best of our knowledge, we present the first work in the direction of automated multimodal detection of online antisemitism. The task poses multiple challenges that include extracting signals across multiple modalities, contextual references, and handling multiple aspects of antisemitism. Unfortunately, there does not exist any publicly available benchmark corpus for this critical task. Hence, we collect and label two datasets with 3,102 and 3,509 social media posts from Twitter and Gab respectively. Further, we present a multimodal deep learning system that detects the presence of antisemitic content and its specific antisemitism category using text and images from posts. We perform an extensive set of experiments on the two datasets to evaluate the efficacy of the proposed system. Finally, we also present a qualitative analysis of our study.